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INTRODUCTION 


1.0 


1.1  Background 

The  increasing  complexities  of  modem  weapon  systems  and  missions  are  placing 
demands  on  pilots  and  aircrews  that  often  exceed  their  ability  to  perform  in  the 
traditional  manner.  Advanced  avionics  and  supporting  ground  systems  are  capable  of 
collecting,  processing  and  distributing  unprecedented  amounts  of  operating  data,  much 
of  which  is  essential  ir  order  to  cope  with  current  and  projected  mission  requirements 
and  threat  situations.  At  the  same  time,  the  air  vehicle,  subsystems,  and  weapons  are 
themselves  becoming  more  sophisticated  in  order  to  support  performance  goals  and 
extended  operating  conditions.  As  a  result,  pilots  are  being  faced  with  vastly  more 
information  to  interpret,  more  complex  instructions  to  give  their  onboard  systems,  and 
considerably  less  time  to  perform  these  functions.  Desired  performance  gains 
associated  with  new  and  "improved"  weapon  systems  may  not  be  achieved  if  these 
requirement*  exceed  the  aircrew's  ability  to  perform. 

Historically,  pilots  have  been  able  to  function  successfully  at  an  "operator"  level, 
exercising  direct  monitoring  and  control  over  many  of  the  individual  components  and 
subsystems  that  comprise  the  total  weapon  system.  In  this  capacity,  the  pilot 
effectively  performed  the  integration  function  in  real-time.  Using  essentially  raw 
data,  he  was  required  to  search,  monitor,  interpret,  transform,  integrate,  and  evaluate 
multiple  readouts  in  order  to  arrive  at  the  alternatives,  decisions  and  control  actions 
needed  to  manage  his  aircraft  and  mission.  Raw  data  for  this  purpose  was  most 
typically  obtained  from  dedicated  electromechanical  instrumentation  in  alpha-numeric 
form.  In  the  current  complex  environment  this  approach  is  no  longer  feasible}  it  has 
become  clear  that  the  information  processing  capacity  of  man  can  severely  limit  the 
overall  performance  of  the  system.  Modem  efforts  toward  cockpit  integration  are 
dramatically  enhancing  the  role  of  the  crew,  allowing  him  to  more  effectively  exercise 
appropriate  aircraft  and  mission  control  functions  at  a  "management"  level. 

Technologies  that  have  resulted  in  increased  complexity  have,  at  the  same  time, 
created  some  of  the  advances  needed  to  solve  the  problem.  Rapid  advances  in 
computing  and  data  processing  technology  have  made  it  possible  to  automate  many  of 


the  raw  data  functions  previously  performed  manually,  thus  offering  the  pilot 
processed  decision-Ir/el  information  tailored  to  "management”  responsibilities.  Mass 
storage  and  high  speed  processing  also  provide  the  potential  for  more  and  better 
systems  and  mission  information  available  to  the  pilot  than  he  could  hope  to  achieve 
manually,  as  well  as  the  means  to  help  determine  what  information  is  needed  and 
when.  Multifunction  electro-optical  displays  and  controls  have  given  the  crewstation 
designer  and  the  pilot  vastly  increased  levels  of  flexibility  in  the  cockpit.  The  flexible 
programming  offered  by  these  devices  allows  for  the  true  integration  of  information 
and  control  functions  according  to  the  needs  of  the  pilot,  and  for  the  rapid 
reconfiguration  of  the  cockpit  based  on  changing  mission  and  system  conditions.  The 
numerous  readouts  of  alpha-numeric  raw  data  once  used  by  the  pilot  can  thus  be 
replaced  by  integrated,  mission-oriented  displays  formatted  for  ease  of  interpretation, 
heightened  situational  awateness,  and  rapid  decision-aiding. 

Until  recently,  the  advantages  of  programmable  electro-optical  displays  and  controls, 
including  the  use  of  color  and  graphic  or  pictorial  information,  have  only  been  partially 
exploited.  Although  the  flexibility  exists,  there  has  only  infrequently  been  a  re¬ 
examination  of  pilot  information  needs  together  with  the  formats  and  symbology  best 
able  to  convey  this  information.  Instead,  there  has  been  a  tendency  to  mimic  the 
information  and  formats  characteristic  of  the  older  electro-mechanical  devices.  The 
goal  of  the  present  Pictorial  Format  Displays  program  has  been  to  extensively  explore 
the  concept  of  replacing  the  alpha-numeric  data  typically  used  in  the  past  with 
integrated  graphic  and  pictorial  display  formats.  In  the  phases  of  the  program 
reported  here,  representative  electronic  pictorial  displays  were  developed,  based  on 
format  concepts  provided  by  the  Government,  and  these  displays  were  then  evaluated 
in  a  real-time,  full-mission  piloted  simulation. 

Sponsorship  of  this  work  was  provided  by  the  Crew  Systems  Development  Branch  of 
the  Air  Force  Flight  Dynamics  Laboratory  (AFWAL/F1GR),  the  ECM  Advanced 
Development  Branch  of  the  Air  Force  Avionics  Laboratory  (AFWAL/AAWD),  the  Air 
Force  Armament  Laboratory  (AFATL/DLJA),  and  the  Naval  Air  Development  Center. 
Both  services  are  conducting  a  comprehensive  interrelated  program  to  develop  the 
advanced  integrated  crewstation  technologies  needed  for  future  aircraft.  This  work 
represents  a  contributing  part  of  these  efforts. 
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1.2 


The  Two  Simulation  Studies 


The  program  reported  here  consisted  of  two  sequential  studies.  The  basic  study  was  a 
general  evaluation  of  the  concept  of  pictorial  format  displays,  while  the  second  used 
the  same  basic  displays  but  concentrated  on  threat  warning  formats  of  the  sort  which 
might  be  employed  in  future  tactical  aircraft.  Sections  2  through  5  and  Appendices  A 
and  B  of  this  report  cover  the  basic  study.  Sections  6  through  10  and  Appendices  C 
and  D  cover  the  threat  warning  study.  Section  1 1  includes  the  combined  conclusions 
and  recommended  format  changes. 

13  Test  Objectives  -  Basic  Study 

The  basic  study  had  three  primary  objectives.  One  of  these  was  to  evaluate  usability 
and  pilot  acceptance  of  a  set  of  service-provided  pictorial  format  concepts  for 
electro-optical  displays.  The  second  objective  was  to  determine  whether  the  degree  of 
usability  and  pilot  acceptance  of  the  pictorial  formats  is  a  function  of  three  basic 
display  presentation  modes:  monochromatic,  color  line,  and  color  fill.  The  third 
objective  was  to  revise  the  formats  based  on  the  data  collected.  All  pictorial 
elements  in  the  monochromatic  formats  were  composed  of  black  and  white  lines  and 
surfaces.  Pictorial  representations  in  the  color  line  formats  were  depicted  by  simple 
outlines,  but  several  different  colors  were  used  to  define  elements  within  the  formats. 
In  the  color  fill  formats,  surfaces  enclosed  by  line  segments  were  also  colored. 

Realistic  man-in- the- loop  simulation  was  used  to  evaluate  the  display  formats.  USAF 
and  Navy  aircrews  evaluated  pictorial  formats  for  seven  basic  types  of  displays:  head- 
up  flight  displays,  head-down  flight  displays,  navigation/tactical  situation  displays, 
system  status  displays  (f uel,- electrical,  hydraulic),  engine  displays,  stores  management 
displays,  and  emergency  procedures  displays.  Data  collected  from  these  studies  were 
used  to  refine  the  original  formats,  where  appropriate,  and  the  results  are  documented 
in  this  report.  Figures  1.3-1  and  1.3-2  show  examples  of  the  formats.  Section  3 
discusses  each  in  detail. 


ELECTRICAL  ADVISORY  ELECTRICAL  STATUS 


2J3  TEST  EQUIPMENT  AND  FACILITIES  -  BASIC  STUDY 

The  simulation  hardware  configuration  which  supported  the  basic  Pictorial  Format 
Display  Evaluation  study  is  shown  in  Figure  2.0-1.  For  discussion,  the  equipment  can 
be  divided  into  four  groups;  simulation  host  computing  and  bus  communication;  digital 
graphics  system;  crew  station  with  displays;  and  supporting  elements,  including  data 
recording  units. 

2.1  Simulation  Hoot  Computers  and  Bus  Communication 

An  advanced  tactical  lighter  model  and  a  navigation  cell  with  real  world  coordinates 
were  used  as  a  baseline  for  both  the  manual  and  autopilot  flight  modes.  Three  Varian 
V-76  computers  and  one  Floating  Point  Systems  AP-120B  array  processor  provided  for 
airplane  modeling,  navigation  cells,  simulation  operating  systems,  instruction  sets  and 
airplane  derivatives  for  graphics  assembly,  and  on-line  data  recording.  This  part  of 
the  simulation  was  recomputed  each  42  milliseconds.  All  available  data  (bidirectional) 
were  transmitted  within  two  computer  frame  times  of  less  than  84  milliseconds. 
Digital  data  was  passed  from  V-76  computer  No.  5  through  a  MIL-STD-1553B  digital 
data  bus  composed  of  one  bus  control  interface  unit,  two  base  band  fiber  optic  modem 
sets  and  two  bus  interface  adapter  units  to  the  crew  station  I/O  and  to  the  graphics 
generators.  This  was  a  1  Mhz,  bidirectional  data  bus.  Programs  were  written  in 
FORTRAN  and  assembly  languages.  Assembly  language  (about  20  percent  of  total 
software)  was  used  for  the  crewstation  I/O,  special  switch  routines,  recording  devices, 
and  the  airplane  equations  of  motion^ 

'  f 

Z2  Digital  Graphics  System 

The  digital  graphics  system  was  supported  by  a  dedicated  SEL  32/2750  computer  which 
had  a  300  megabyte  (MB)  memory  disc,  800/1600  bpi  1/2  inch  magnetic  tape  unit,  high 
speed  line  printer,  and  several  line  terminals.  Two  high  speed  devices  served  as 
parallel  interfaces  between  the  SEL  32/2750  computer  and  two  graphics  generators. 

One  Megatek  7250  color  raster  graphics  generator  provided  RGB  video  outputs  for 
both  the  HUD  and  the  VSD  displays.  Both  display  channels  had  512  by  512  pixels 


Figum  2.0-1.  Simulation  Configuration 


resolution  updated  at  30  frames  per  second  interlaced.  With  the  32  bit  parallel 
inter tace  to  the  Selbus,  (3  MB/sec),  speeds  were  1.75  microseconds  per  vector  and  160 
nanoseconds  per  pixel  average  writing  time.  Internal  hardware  architecture  allowed 
for  color  block  fill,  vectoring,  and  2D/3D  transformations.  The  32  bit  wide  bus 
allowed  up  to  two  vectors  to  be  processed  with  a  single  memory  access,  thus 
minimizing  processing  time  and  maximizing  throughput  to  the  vector  generator  and 
raster  displays.  The  hardware  transformation  modules  scaled,  rotated,  and  translated 
each  vector  and  then  clipped  the  result  to  the  specified  viewport,  this  transformation 
was  performed  on  each  refresh  Cycle,  providing  the  fastest  frame  update  rate  possible. 
The  Megatek  7250  was  supported  by  WAND  7200,  a  modular  software  package 
following  guidelines  derived  from  the  latest  efforts  in  computer  graphics 
standardization.  Call-up  routines  resided  in  the  SEL  32/2750. 

One  Lexidata  3400  color  raster  graphics  generator  provided  RGB  video  output  for  the 
HSD.  Line  resolution  was  640  by  512  pixels  updated  at  ,30  frames  per  second 
interlaced.  It  was  interfaced  through  a  16  bit  parallel  bus  to  the  SEL  computer.  Burst 
data  transfer  rates  of  up  to  1  megaword  per  second  (16  bit  words)  were  typieaL 
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Normal  operating  speeds  were  2  microseconds  per  pixel  update  and  3  microseconds  per 
vector  with  a  13  microsecond  vector  set-up  time.  Image  control  and  manipulation 
features  consisted  of  high  speed  color  look-up  tables,  zoom  control  with 
magnifications  from  1  to  16  power  in  integer  increments,  pan  controllers,  multiple 
overlays,  blink  and  circle  controllers.  Image  manipulation  subroutines  were  stored  in 
the  Lexidata  3400  either  in  PROM  or  user  programmed  RAM  and  called  from 
FORTRAN  library  programs  residing  in  the  SEL  32/2750. 

Two  Discovision  Model  PR-7820-3  video  disk  units  provided  525  line,  30  frame  per 
second,  NTSC  video  through  two  Lenco  PCD-363  demodulator  color  units  to  the  two 
MPDs.  The  Lenco  decoders  received  composite  color  video  from  the  disk  players, 
demodulated  color  and  stripped  RGB  and  sync  from  the  signal.  This  allowed  RGB 
displays  and  monitors  to  be  used.  The  MPD  units  had  limiting  resolution  of 
approximately  300  TVL.  The  PR-7820-3  is  an  optical  laser  video  disk  player  with  a 
storage  capacity  of  up  to  54,000  frames  (1  image  list  per  frame)  per  side.  Direct 
random  access  to  any  frame  out  of  54,000  is  possible  within  2.5  seconds.  Access 
speeds  of  60-120  milliseconds  were  realized  by  selective  programming  of  the  image 
lists  onto  the  disk.  The  disk  units  were  interfaced  through  two  RS-232  ports  of  the 
crew  station  I/O  bus.  Digital  instructions  directed  the  disk  units  for  freeze  frame, 
search,  frame-by-frame,  forward,  reverse,  and  frame  by  number  operations. 

2.3  Crew  Station  and  Displays 

The  front  seat  of  the  multimission  two-man  tandem  crew  station  was  modified  and 
used  ir.  support  of  this  program  as  represented  in  Figure  2.3-1  and  pictured  in  Figure 
2.3-2.  The  crew  station  was  interfaced  through  a  MIL-STD-1553B  bus  interface 
adapter  to  the  1553B  IMhz  data  bus.  Bidirectional  digital  data  was  converted  in  the 
crew  station  I/O  into  discrete  in  and  out,  analog  to  digital,  digital  to  analog,  and 
digital  to  synchro  data.  A  local  Motorola  68Q00  microprocessor,  connected  to  the  I/O 
16  bit  parallel  bus,  provided  for  local  intelligence  (i.e,  instrument  scaling,  diagnostics, 
test  routines,  and  demonstration  capability). 

Five  color  CRT  monitors  were  installed  in  the  cockpit  for  this  simulation.  The  HUD,. 
VSD  and  HSD  were  eight-inch  diagonal,  modified  NEC  narrow  shadow  mask  monitors. 
They  had  approximately  480  lines  of  vertical  resolution.  The  HUD  was  designed  for  a 
23-degree  horizontal  by  17-degree  vertical  field  of  view  and  was  focused  at  near 


HUD  • 


MASTER  CAUTION 


infinity.  The  VSD  monitor  was  underscanned  to  provide  a  square  format.  The  HSD  had 
a  horizontally  oriented  4  by  3  aspect  ratio. 

The  MPD's  were  five-inch  diagonal  modified  Hitachi. monitors.'  The  active  display  area 
was  3-1/4  by  4-1/4  inches  with  200  by  280  color  triads.  The  left  MPD  was  oriented  «ind 
scanned  horizontally  and  the  right  MPD  was  oriented  and  scanned  vertically. 

In  addition  to  the  full  color  CRT  displays,  a  flight  control  stick,  dual  throttles,  an 
altimeter,  accelerometer,  attitude  indicator,  vertical  speed  indicator,  angle-of-atta:k, 
and  Mach-airspeed  instruments  were  installed  and  available  for  use.  An  autopilot 
comprising  bedding  hold,  nav  steer,  altitude  hold,  and  speed  hold  were  available  as  well 
as  a  selection  of  controllers,  multifunction  switches,  and  subsystem  control  panels. 

2.4  Supporting  Elements 

The  simulator  crew  station  was  placed  before  a  60  degree  vertical  x  160  degree 
horizontal  spherical  section  screen.  The  screen  had  a  radius  of  15  feet  with  a  high 
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Figure  2.3-2.  Cockpit  Overall  View 


gain  coating.  A  70  millimeter,  variable  speed  motion  picture  projector  operating  in 
coincidence  with  the  air-to-ground  simulation  scenario  presented  the  out-of-cockpit 
scene.  Symbology  on  the  HUD  was  superimposed  on,  and  tracked,  the  out-of-cockpit 
scene,  thus  providing  a  basis  for  color  observation  evaluations. 

Oculometer  data  was  obtained  with  an  NAC  Model  4  eye  mark  recorder,  a  device  that 
enables  the  point  of  visual  regard  to  be  monitored.  It  was  lightweight  (14.4  oz), 
required  minimal  time  to  set  up  and  calibrate,  and  allowed  freedom  of  head 
movement.  A  V-shaped  reticle  was  superimposed  on  a  visual  image  of  the  cockpit  over 
a  range  of  approximately  60  degrees  horizontally  by  40  degrees  vertically.  The  reticle 
and  image  were  conveyed  via  a  fiber  optic  link  to  a  Dage-650  silicon  intensified  tube 
(SIT)  525  line  video  camera.  The  combined  image  was  then  recorded  on  a  JVC  model 
CR  6600U  video  tape  recorder.  For  runs  when  the  eye  mark  recorder  was  not  used,  a 
second  Dage  camera  was  employed  to  record  pilot  cockpit  activity.  A  Portae  Model 
DS-1  time  and  window  generator  were  used  to  key  and  mix  digital  clock  time  and 
frame  number  information  on  each  frame  of  the  imagery.  This  allowed  rapid  access  to 
specific  frames  of  interest.  The  video  mixing  unit  clock  was  activated  by  a  command 
from  the  Varian  V-76  host  computer  at  the  start  of  the  simulation  and  then  ran  in 
parallel  with  the  computer  clock.  The  maximum  discrepancy  between  times  recorded 
on  the  video  imagery  and  event  times  recorded  in  the  V-76  simulation  computer  was 
approximately  50  milliseconds  over  the  course  of  a  30-minute  simulation.  A 
monochrome  TV  monitor  was  on-line  to  continuously  monitor  the  final  video  being 
recorded  on  the  JVC  video  recorder.  The  pilot’s  verbal  reports  were  recorded  on  the 
recorder  audio  channel. 


A  test  conductor's  console  was  positioned  just  aft  and  above  the  crew  station.  The 
test  conductor  had  access  to  the  host  computer  controls.  He  monitored  the  simulation 
run  variables  via  a  TV  repeater  from  the  V-76  computer  console.  He  was  in  constant 
communication  with  the  subject  as  well  as  with  other  personnel  operating  the 
simulation  equipment. 


3.0 


TfcST  FORMATS  -  BASIC  STUDY 


All  the  formats  tested  in  this  evaluation  -  the  Head-Up  Display  (HUD),  the  Vertical 
Situation  Display  (VSD),  the  Horizontal  Situation  Display  (HSD)  and  the  systems 
formats  -  were  created  in  three  versions  or  display  presentation  modes.  The  color  fill 
mode  represented  a  color  hybrid  or  raster  display  system.  Formats  were  created  with 
either  lines  or  filled  areas  of  color.  The  color  line  mode  represented  a  color  stroke  or 
calligraphic  display  system.  The  formats  were  created  from  colored  lines.  In  the 
monochrome  mode,  lines  and  filled  areas  were  used  but  in  w!iite-on-black,  with  no 
color. 

Many  of  the  format  concepts  tested  in  this  evaluation  were  initially  developed  in  an 
earlier  program,  and  were  reported  in  Reference  1.  A  number  of  formats  were 
changed  only  as  necessary  to  implement  them  in  this  simulation.  In  all  cases,  the 
salient  features  of  the  earlier  conceptual  formats  were  retained. 

Considerable  effort  was  expended  designing  and  detailing  the  formats  of  Reference  1 
for  dynamic  display  on  CRT’s  in  a  simulator  cockpit  and  further  tailoring  them  to 
support  a  representative  mission  scenario.  All  the  displays  were  responsive  to  both  the 
mission  and  the  pilots'  actions.  The  HUD,  VSD  and  HSD  were  programmed  to  fly  the 
mission  with  real-time  updates  at  an  acceptable  rate.  They  were,  literally,  computer¬ 
generated  imagery.  The  formats  which  appeared  on  the  MPD's  were  individually 
static,  but  enpugh  of  them  (834)  wnre  created  to  represent  all  the  conditions  and 
system  states  required  by  the  mission.  The  stores  programming  format  was  a  new 
development,  created  to  provide  control  necessary  to  exercise  the  stores  status 
format. 

3.1  Head-up  Display 

The  head-up  display  (HUD),  shown  in  Figure  3.1-1,  featured  a  pathway-in-the-sky 
flight  director,  an  airplane  symbol,  synthetic  terrain  information  and  data  boxes  with 
digital  airspeed,  heading  and  altitude.  It  was  the  primary  flight  display,  and  symbology 
was  true-scale  with  the  outside  world.  The  HUD  field  of  view  was  23  degrees  across 


Figure  3.1-1.  HUD  Display  TF/TA  Format 


by  17  degrees  high.  Artificial  terrain,  the  horizon  and  the  pathway  were  ground- 
stabilized. 

The  pathway  represented  the  desired  flight  path.  The  entrance  gate  represented  an 
area  300  feet  wide  by  130  feet  high.  It  was  6000  feet  tang,  tapered  in  perspective 
toward  the  far  end,  and  had  a  single  line  extension  20,000  feet  long.  There  were 
internal  lines  on  the  floor  and  sides  of  the  pathway  at  10QG  foot  intervals.  Numbered 
flags  on  the  side  of  the  pathway  represented  waypoints  or  ground  targets. 

At  high  altitude,  the  terrain  was  replaced  by  an  artificial  horizon  line.  During  flight,  a 
caution  or  warning  condition  was  indicated  by  color  change  (to  yellow  or  red, 
respectively)  of  the  airplane  symbol  and  data  blocks.  The  symbology  flashed  to 
indicate  caution  or  warning  in  the  monochrome  mode.  A  red  (or  white  for 
monochrome)  "X"  symbol  appeared  in  the  center  of  the  airplane  symbol  for  both 
air-to-air  and  air-to-grcund  weapon  delivery.  It  came  on  solid  to  indicate  entry  into 
the  delivery  envelope,  flashed  when  the  pilot  pressed  his  weapon  delivery  handof f 
button  and  disappeared  when  the  weapon  was  released  or  the  envelope  had  passed. 

In  the  color  fill  mode,  mountains  on  the  HUD  were  green,  the  ground  plane  was  a 
darker  green,  the  channel  was  white  with,  black  markings  and  the  airplane  symbol  was 
blue.  In  color  ILnt  mode,  the  terrain  shapes  were  outlined  in  green  and  the  channel  was 
outlined  in  white.  In  the  monochrome  mode,  the  terrain  shapes  and  channel  were  black 
with  white  outline.  In  both  color  line  and  monochrome  modes,  radial  lines  on  the 
ground  plane,  originating  at  the  center  of  the  horizon,  provided  sky -ground 
differentiation. 

X2  Vertical  Situation  Display 

At  high  altitude,  the  vertical  situation  display  (VSD)  had  conventional  attitude  director 
Indicator  symbology.  Below  an  arbitrarily  selected  10,000  feet,  the  low  altitude  or 
terrain-foliowing/terrain  avoidance  VSD  format  was  available. 

The  low  altitude  VSD  format  (Figure  3.2-1)  gave  a  perspective  view  of  the  aircraft, 
the  desired  flight  path,  the  terrain  and  surface  threats.  A  90-degree  wide  by  67- 
degree  high  field  of  view  wes  shown  from  a  viewpoint  6000  feet  behind  and  1000  feet 


Figure  3.2-1.  VSD  Display  TF/TA  Format  -  Color  Fill 


above  the  aircraft.  The  desired  flight  path  was  portrayed  as  a  five  mile  long  ribbon 
beginning  1000  feet  ahead  of  the.  aircraft  symbol  and  ending  20,000  feet  ahead.  It  was 
roughly  comparable  to  the  floor  of  the  HUD  pathway.  Like  the  HUD  pathway,  the  VSD 
ribbon  had  numbered  flags  representing  waypoints  and  targets. 

Terrain,  in  the  low-altitude  VSD,  was  represented  as  three  dimensional  mountain 
shapes,  brown  above  current  aircraft  altitude  and  green  below.  A  grid  on  the  ground 
plane  helped  create  the  perspective  effect.  Rivers,  roads,  targets  and  the  Forward 
Edge  of  the  Battle  Area  (FEBA)  were  shown  on  the  ground  plane. 

Surface-to-air  missile  (SAM)  and  anti-aircraft  artillery  (AAA)  threats  were  shown  on 
the  VSD  as  three  dimensional  shapes,  colored  yellow  and  red  to  indicate  degree  of 
lethality.  The  SAMs  were  roughly  conical  with  apex  downward  and  axis  inclined 
slightly  toward  the  planned  flight  route.  The  outer  volume  of  each  SAM  shape  was 
yellow  and  the  inside  red.  Whenever  two  SAM  volumes  intersected,  lethality  was 
considered  to  be  high  and  the  intersection  was  red.  AAA  sites  were  shorter  truncated 
cones  (base  down)  with  uniformly  high  lethality,  and  thus  were  solid  red. 

In  color  line  mode  (Figure  3.2-2),  the  mountains  and  threats  on  the  VSD  were  outlined 
with  the  appropriate  color— brown  or  green  for  mountains  and  yellow  or  red  for 
threats.  In  monochrome  mode  (Figure  3.2-3),  mountains  above  current  altitude  and  the 
high  lethality  portions  of  threat  shapes  were  solid  white  with  the  remainder  of 
mountains  and  the  threats  outlined  in  white. 

L3  ,  Horizontal  Situation  Display 

The  horizontal  situation  display  (HSD)  was  a  plan  view  projection  of  situation  and 
tactical  information.  It  was  a  track-u  >  format  centered  on  the  airplane  symbol  as 
shown  in  Figure  3.3-1.  Information  on  the  HSD  included  the  planned  flight  route  with 
numbered  waypoints  and  targets,  mountains,  threats,  rivers,  roads  and  the  FEBA. 

In  the  low  altitude  mode,  available  below  10,000  feet,  these  features  were  shown 
against  a  green  background.  As  in  the  VSD,  mountains  were  brown  above  current 
aircraft  lltitude.  SAM  and  AAA  threats  were  shown  as  current  altitude  slices  through 
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Figure  3.2-2.  VSD  Display  TF/TA  Format  -  Color  Line 
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3-i.  HSD  Display  TF/TA  Format  With  Target  Inset 


the  red  and  yellow  lethality  shapes  described  above.  Digital  heading  was  given  in  a 
box  at  the  top  center  of  the  display. 

The  high  altitude  HSD  showed  all  the  above  information  except  threats  and  mountains. 
The  background  was  black.  In  air-to-air  mode,  the  HSD  contained  only  the  symbology 
shown  in  Figure  3.3-2.  The  sec*or  shapes  represented  low  (outer)  and  high  (inner) 
probability  of  kill  areas  for  both  own  and  adversary  aircraft.  In  this  simulation,  only 
the  80-mile  scale  was  available  for  air-to-air  mode  and  the  range  rings  had  20  and  40 
nautical  mile  radii. 

A  number  of  option  switches  were  available  for  the  HSD.  "Range  Increase"  and 
"Range  Decrease"  allowed  the  pilot  to  select  amoung  40,  80,  160  and  320  nautical  mile 
along-track  display  ranges.  The  selected  range  was  given  in  a  box  in  the  lower  left 
comer  of  the  HSD.  "Fuel  Range"  allowed  the  pilot  to  select  or  deselect  two  range 
rings.  They  represented  normal  and  extended  fuel  range  limits.  "Target"  put  an 
expanded  inset  of  the  target  area  on  the  right  side  of  the  display.  A  line  led  lrom  this 
target  inset  to  the  actual  target  location.  Three  other  buttons  allowed  the  pilot  to 
display  digital  time  and  distance  to  the  home  base,  the  next  target  or  the  next 
waypoint.  This  information,  when  selected,  was  displayed  at  the  lower  center  and 
right  of  the  HSD. 

3.4  Multipurpose  Displays 

Formats  on  the  multipurpose  displays  pictorially  represented  airplane  systems: 
engines,  stores,  electrical,  hydraulic  and  fuel.  Each  of  these  systems  had  a  status 
display.  Several  (engines,  electrical  and  hydraulic)  had  associated  advisory  displays, 
and  there  was  also  a  programming  display  for  the  stores  system.  The  engine  status, 
electrical  status  and  hydraulic  status  formats  were  assigned  to  the  right  MPD.  The 
two  stores  formats,  the  fuel  status  format  and  the  advisory  formats  were  on  the  left 
MPD.  Each  of  these  formats  is  discussed  In  more  detail  below. 

Like  the  flight  and  situation  displays,  these  systems  formats  each  had  three  versions, 
following  the  rules  for  color  fill,  color  line  and  monochrome  described  in  paragraph 
3.0. 
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3.4.1 


Engine  Formats 


Figure  3.4.1-1  shows  the  engine  status  format  for  a  two-engine  airplane.  Engine  body 
outlines  were  given  with  important  engine  parameters  depicted  symbolically.  The 
articulating  inlet  and  exhaust  doors  were  shown.  Turbine  inlet  temperature  (TIT)  and 
exhaust  gas  temperature  (EGT)  were  shown  as  thermometers  with  safety  limits 
represented  by  horizontal  bars.  Compressor  speed  (Nl)  and  turbine  speed  (N2)  were 
shown  as  vertical  bars  with  limits.  Fuel  flow  (FF)  to  each  engine  was  shown  as  an 
arrow  with  moving  bubbles  and  a  bar  for  fuel  cut-off.  The  size  of  the  arrows 
represented  rate  of  fuel  flow.  The  flame  at  the  rear  of  each  engine  changed  color  and 
moved  outside  the  burner  section  to  indicate  afterburner  operation.  Figure  3.4. 1-2 
shows  a  series  of  engine  advisory  formats  directing  the  pilot  through  remedial  steps 
after  an  engine  fire.  The  pilot  was  advised  by  these  formats  to  (a)  bring  the  throttle 
of  the  affected  engine  to  idle,  (b)  cut  off  the  fuel  to  that  engine,  (c)  release  the  fire 
extinguishing  agent  and,  (d)  with  the  fire  extinguished,  land  as  soon  as  possible. 

3.4.2  Stores  Formats 

The  Stores  Status  format  showed  presence  or  absence  of  each  store  on  the  aircraft  and 
the  state  of  the  stores  selection  and  stores  programming  options.  Figure  3.4.2-1  is  an 
example  of  one  configuration  of  the  stores  status  format.  Represented  are  two  ECM 
pods  on  the  wing  tips,  two  napalm  canisters,  twelve  Mark  82  bombs,,  and  two  air-to-air 
missiles.  The  weapons  shown  in  dashed  outline  are  on  board  but  not  selected.  Eight  of 
the  bombs  are  selected  with  nose  and  tail  c  olored  green  to  indicate  programmed 
fusing.  The  center  sections  of  the  selected  bombs  are  yellow.  These  would  appear 
green  if  the  Master  Arm  switch  were  turned  On  and  the  bombs  were  completely  ready 
to  drop.  The  triangles  near  the  forward  two  bombs  indicate  that  they  will  be  the  first 
two  to  drop.  If  any  store  had  already  been  expended;  that  position  would  be  vacant. 

The  wingtip  ECM  pods  were  represented  to  have  only  three  states— off,  standby  and 
on.  When  off,  they  appeared  with  dashed  outline.  In  standby,  the  ECM  pod  symbol 
turned  yellow  and  had  a  solid  outline.  When  the  ECM  was  turned  on,  the  symbol  turned 
blue  and  yellow  lighting  bolts  appeared  forward  of  the  pod  symbol.  ECM  functioned 
separately  from  dropable  stores,  and  could  be  selected  in  combination  with  any  of  the 
weapons  options. 
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The  only  malfunction  in  the  stores  system  was  a  single  hung  bomb  after  the  bomb  drop. 
When  that  happened,  the  format  was  bordered  in  red,  the  remaining  (hung)  bomb 
turned  red  and  the  word  "HUNG”  was  written  alongside  the  bomb.  When  the  pilot 
safed  the  fuses,  the  nose  and  tail  of  the  bombs  went  to  outline  red.  When  he  turned 
the  master  arm  switch  off,  the  belly  went  to  outline  red  and  when  he  deselected 
bombs,  the  border  turned  from  red  to  yellow. 

For  programming  the  stores,  pilots  in  this  evaluation  used  alphanumeric  menu  lists. 
One  such  list  was  available  for  each  type  of  store  and  contained  all  the  available 
options  for  the  selected  store.  Figure  3.4.2-2  is  an  example  of  the  Stores  Program 
format.  In  this  example,  eight  Mark  82  bombs  were  selected,  to  be  delivered  in  pairs 
at  150  foot  intervals  with  nose  and  tail  fused.  The  pilot  could  reprogram  any  store 
using  the  select  and  program  buttons  located  just  above  and  below  the  Stores  Program 
format. 

3.4.3  Electrical  Formats 

The  electrical  system  was  represented  by  two  formats.  One,  the  Electrical  Status 
format  shown  in  Figure  3.4.3-1  was  a  simplified  schematic  of  the  system.  If  some 
element  of  the  electrical  system  failed,  a  red  (for  permanent  failure)  or  a  yellow  (for 
temporary  failure)  border  appeared  around  the  format  and  the  picture  of  the  failed 
element  changed  character.  The  failed  element  was  shown  against  a  white  background 
and  a  red  X  was  drawn  through  it,  closed  for  permanent  failure  and  open  for 
temporary.  The  Electrical  Advisory  format  showed  required  actions  in  pictorial  form. 
Figure  3.4.3-2  indicates  an  example  of  a  permanent  failure  of  the  right  generator.  The 
advisory  is  to  turn  the  right  generator  switch  off  and  to  land  as  soon  as  possible. .  . 

3.4.4  Hydraulic  Formats 

The  Hydraulic  Status  format,  an  example  of  which  is  shown  in  Figure  3.4.4-1,  followed 
a  non-schematic  pictorial  philosophy.  The  format  was  a  matrix  of  system  (IA,  IB,  2A, 
2B)  by  status  (normal,  temporary  failure,  permanent  failure).  In  case  of  failure,  the 
symbol  representing  the  failed  element  moved  to  the  second  column  (temporary 
failure)  or  the  third  column  (permanent  failure).  In  addition,  a  yellow  border  appeared 
around  both  the  whole  format  and  the  failed  element  for  permanent  failure.  It  was 


Figure  3.4. 3-2.  Electrical  System  Advisory  Format 


assumed  that  the  hydraulic  system  had  enough  redundancy  that  a  temporary  failure  did 
not  warrant  caution  status. 

The  Hydraulic  Advisory  formats  did  not  portray  remedial  action,  but  instead  indicated 
the  affected  airplane  system.  Figure  3.4. 4-2  shows  the  example  of  a  permanent 
failure  of  the  IB  hydraulic  system  and  indicates  that  the  port  side  control  surfaces  are 
affected. 

3.4-5  Fuel  System  Format 

The  fuel  system  was  shown  against  an  airplane  outline  in  the  Fuel  Status  format.  A 
five-tank  system  was  shown  with  valves,  crossfeed,  a  refueling  probe  and  a  single 
boost  pump.  It  was  assumed  that  additional  details  would  be  provided  on  an  advisory 
display  as  needed.  The  level  of  fuel  in  each  tank,  the  state  of  each  valve  and  the  boost 
pump  were  represented.  Figure  3.4.5-1  shows  roughly  35  percent  of  the  fuel  remaining 
and  a  closed  valve  on  the  right  wing  tank  resulting  in  a  fuel  imbalance. 
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Hydraulic  System  Advisory  Format 


Fuel  System  Status  Format 


TEST  PLAN  -  BASIC  STUDY 


4.0 

The  test  plan  for  the  basic  study  was  derived  from  the  prime  objectives.  They  were  to 
evaluate  the  usability  and  pilot  acceptance  of  these  pictorial  formats;  determine 
whether  usability  and  pilot  acceptance  are  a  function  of  presentation  mode  - 
monochrome,  color  line  and  color  fill;  and  revise  the  formats  based  on  study  results. 


The  intent  was  to  provide  an  operationally  valid  evaluation  under  conditions  of 
moderate  experimental  control.  To  this  end,  current  military  fighter  or  attack  pilots 
were  recruited  as  evaluators  and  a  within-subjects  factorial  design  was  selected.  The 
test  procedure  provided  an  intensive  training  and  practice  period  before  testing  began. 
The  test  scenario  exposed  the  salient  features  of  the  formats.  Data  were  collected  to 
support  responses  to  the  program  objectives. 

4.1  Test  Subjects 

A  total  of  eighteen  (IS)  pilots  served  as  subjects  in  the  study.  Nine  pilots  each  were 
obtained  from  McChord  AFB  and  from  Whidbey  Island  NAS.  The  Air  Defense 
Command  pilots  from  McChord  were  current  in  the  F-106  airplane  or  the  T-33  while 
the  tactical  fighter  pilots  from  Whidbey  were  current  in  either  the  A-6  or  the  A-7. 
the  special  qualifications  of  the  pilots  are  listed  in  Table  4 .1-1.  The  pilots  had  an 
average  age  of  29  years  and  an  average  of  1,310  flying  hours  in  a  variety  of  jet 
aircraft.  Two  additional  pilots,  one  from  each  command,  served  as  check-out  pilots. 
Th*  y  flew  the  same  schedule  as  the  other  pilots.  The  simulation,  but  not  the  formats, 
changed  in  significant  ways  after  their  participation.  For  this  reason,  their 
performance  data  are  not  included  but  their  responses  to  the  usability  questionnaire 
and  the  general  questions  are  included. 

Na'  -'es,  telephone  numbers,  and  home  organizations  for  all  participating  pilots  were 
mac£  available  to  Boeing  approximately  two  weeks  prior  to  the  scheduled  test  date.  A 
training  package  (Reference  2),  designed  to  familiarize  subjects  with  the  test 
objectives,  conditions  of  test  and  the  simulation  facility,  was  forwarded  to  all  pilots 
for  review  prior  to  the  simulation. 
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Table  4. 1 - 1.  Pilot  Qualifications 


Mean 

5228* 

Age: 

29 

24  -  38  yrs 

Total  jat  hours: 

1,510 

350  -  3,800  hrs 

Year  of  pilot  rating: 

1977 

1968-1982 

Operational  experience: 


Branch 

Type  of  aircraft 

N 

A-7,  A -4,  T-2 

AF 

T-38,  T-37,  T-33 

N 

A-6,  A-4,  AF-111,  F-4,  T-38 

AF 

T-38.  T-37,  T-33 

AF 

F-106,  T-38,  T-37,  T-33 

N 

A-6.  A-4.  T-38.  T-2 

N 

A-6.  EA-6B,  T-38 

AF 

F-106,  T-38.  T-37,  T-33 

AF 

F-106.  T-37.  T-38 

N 

A-6,  A-4,  T-2 

AF 

F-106,  T-38.  T-37,  T-33 

N 

A-6,  A-4,  T-28,  T-2 

N 

A-6,  A-4,  T-34B,  T-2 

AF 

T-38.  T-37,  T-33 

AF 

T-38,  T-37,  T-33 

N 

A-6,  TA-4,  T-2 

N 

A-6r  A -68,  TA-4,  T-2 

AF 

T-38,  T-37,  T-33 

ip' 

Ml' 


/ 
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4.2  Test  Design 

The  experimental  design  selected  for  this  study  was  determined,  to  a  large  extent,  by 
two  important  considerations.  The  most  significant  consideration  was  the  requirement 
to  use  experienced  pilots  as  subjects.  Using  experienced  pilots  offers  the  advantage  of 
reducing  the  amount  of  training  required  and  also  maximizes  generalization  of 
simulation  results  to  the  large  population  of  operational  pilots.  On  the  other  hand,  use 
of  experienced  pilots  imposes  a  serious  restriction  on  the  number  of  subjects  available 
for  participation  it:  the  study.  Therefore,  the  experimental  design  selected  for  this 
study  had  to  be  inherently  economical  in  terms  of  subject  utilization. 

A  second  consideration  involved  the  magnitude  of  the  error  variance  components  in 
the  performance  measures.  In  some  experimental  designs  (independent-groups 
designs),  as  the  number  of  test  subjects  decreases,  there  is  an  increasing  risk  that 
individual  differences  between  subjects  may  contribute  to  performance  differences 
between  two  or  more  treatment  conditions.  Since  the  number  of  subjects  available  for 
this  study  was  quite  small,  it  was  decided  to  select  a  within-subjects  experimental 
design  that  would  minimize  effects  due  to  individual  differences. 

A  block  diagram  of  the  test  design  for  the  proposed  study  is  shown  in  Figure  4.2-1.  As 
can  be  seen,  there  were  two  independent  variables:  display  presentation  mode,  and 
eye  movement  recorder.  The  display  presentation  mode  variable  had  three  levels: 
monochromatic,  color  line,  and  color  fill.  Display  presentation  mode  was  selected  as 
an  independent  variable  in  order  to  determine  whether  the  use  of  more  expensive  color 
displays  would  be  justified  by  increased  usability  and  pilot  acceptance. 

The  eye  movement  recorder  variable  had  two  levels:  present  and  absent.  This  device 
was  used  to  obtain  information  about  the  nature  of  the  pilots'  scan  patterns  over  the 
advanced  display  formats.  It  was  selected  as  an  independent  variable  because  the  eye 
movement  recorder  is  an  obtrusive  measurement  device.  Whenever  such  devices  are 
introduced  into  a  realistic  working  environment,  it  is  possible  that  the  subjects  may 
modify  their  typical  behavior.  A  comparison  of  operator  performance  with  and 
without  the  device  was  made  in  order  to  determine  if  the  device,  in  and  of  itself, 
altered  the  subjects'  normal  behavior. 
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MONOCHROMATIC  COLOR  LINE  COLOR  FILL 

DISPLAY  PRESENTATION  MODE 

Figure  4.2-1.  Experimental  Design 

The  six  ceils  in  Figure  4.2-1  represent  a  3  by  2  factorial  design  with  repeated 
measures.  Each  subject  was  tested  in  all  treatment  conditions  of  the  experiment. 
Therefore,  error  variability  due  to  individual  differences  between  groups  of  subjects 
did  not  differentially  affect  the  performance  measures.  This  design  also  minimized 
the  number  of  subjects  required  and  provided  more  data  per  subject  than  any  other 
design.  Training  requirements  were  minimized  since  skills  learned  under  initial 
training  trials  transferred  to  all  subsequent  conditions. 

43  Test  Procedure 


MOVEMENT 

RECORDER 


Before  reporting,  pilots  had  access  to,  and  were  asked  to  study,  a  "flight  manual" 
(Reference  2)  which  described  the  simulator  cockpit,  the  formats  to  be  evaluated  and 
the  missions  to  be  flown.  The  test  subjects  reported  to  the  Kent  Space  Center  in 
pairs.  Most  pairs  consisted  of  one  Navy  and  one  Air  Force  pilot.  Two  days  were 
required  to  process  each  pair  of  subjects  through  the  entire  training  and  test  sequence. 
The  daily  test  schedule  shown  in  Figure  4.3-1  presents  the  sequence  of  events  for  the 
first  two  pilots.  The  same  basic  sequence  of  events  was  used  for  all  subsequent  test 
days  except  that  the  order  of  the  treatment  conditions  was  varied. 

Each  test  day  began  by  bringing  the  simulator  on-line  and  checking  to  see  that  all 
equipment  was  operating  properly.  Any  detected  deficiencies  were  corrected  before 
the  first  practice  or  test  trial  began. 

On  the  first  day,  pilots  were  given  an  introduction  to  the  multimission  simulator  while 
the  cab  was  being  initialized  for  the  first  practice  trial.  Following  the  introduction, 
the  pilots  were  briefed  on  the  general  cockpit  layout,  the  formats  to  be  evaluated,  the 


DAY  1 


0730-1000  Bring  simulator  on  line  and  checkout 
0800-0930  Introduction  to  facility  and  pre-flight  briefing 


0930-1000  Cockpit  familiarization 

1000-1040  Practice  trial,  color  fill,  autopilot,  no  eye  recorder,  no  visual  scene,  subject  1 
1040-1120  Practice  trial,  color  fill,  autopilot,  no  eye  recorder,  no  visual  scene,  subject  2 
1120-1200  Practice  trial,  color  fill,  no  eye  recorder,  no  visual  scene,  subject  1 
1200-1300  Lunch  Break 

1300-1340  Practice  trial,  color  fill,  no  eye  recorder,  no  visual  scene,  subject  2 
1340-1420  Practice  trial,  color  line,  no  eye  recorder,  visual  scene,  subject  1 
1420r1500  Practice  trial,  color  line,  no  eye  recorder,  visual  scene,  subject  2 
1500-1540  Practice  trial,  monochrome,  eye  recorder,  no  visual  scene,  subject  1 
1540-1620  Practice  trial,  monochrome,  eye  recorder,  no  visual  scene,  subject  2 
1620-1700  Test  trial ,  monochrome,  eye  recorder,  subject  1 
1700-1 740  Test  trial,  monochrome,  no  eye  recorder,  subject  2 


DAY  2 

0730-0830  Bring  simulator  on  line  and  checkout 
0830-0910  Test  trial,  color  line,  eye  recorder,  subject  1 
0910-0950  Test  trial,  color  line,  no  eye  recorder,  subject  2 
0950-1030  Test  trial,  color  till,  eye  recorder,  subject  1 
1030-1 1 10  Test  trial,  color  fill,  no  eye  recorder,  subject  2 
1110-1150  Test  trial,  monochrome,  no  eye  recorder,  subject  1 
1150-1300  Lunch  Break 

1300-1340  Test  trial,  monochrome,  eye  recorder,  subject  2 
1340-1420  Test  trial,  color  line,  no  eye  recorder,  subject  1 
1420-1500  Test  trial,  color  line,  eye  recorder,  subject  2 
1500-1540  Test  trial,  color  fill,  no  eye  recorder,  subject  1 
1540-1620  Test  trial,  coior  fill,  eye  recorder,  subject  2 
1540-1700  Post-flight  debriefing 


Figure  4.3- 1.  Daily  Test  Schedule 


operational  procedures  to  be  used,  and  the  flight  plan  for  the  simulated  mission.  A 
one-half  scale  foamcore  mock-up  of  the  simulator  cab  was  used  during  the  cockpit 
layout  briefing.  Static  pictorial  representations  on  film  transparencies  were  used  to 
introduce  display  formats  that  would  be  encountered  during  the  practice  and  test 
trials.  The  briefing  was  conducted  in  a  room  removed  from  the  simulator  cockpit. 
Then  the  pilots  were  taken  to  the  simulator  cab  for  familiarization  with  the  operation 
of  required  cockpit  controls  and  to  calibrate  the  eye  movement  recorder. 


Each  pilot  flew  one  familiarization  trial  and  three  practice  trials,  one  with  each  of  the 
three  presentation  modes.  Familiarization  and  practice  trials  included  the  same  basic 
misson  profile  and  flight  duration  used  in  the  test  trials.  At  the  beginning  of  each 


practice  and  test  trial,  the  pilot  completed  the  preflight  checklist  given  in  Figure 
4.3-2.  The  eye  movement  recorder  was  worn  on  one  practice  trial  to  familiarize  the 
subjects  with  its  characteristics  prior  to  data  collection.  The  VMC  flight  segment  was 
also  presented  during  one  of  the  practice  trials.  After  the  practice  trials,  each  pilot 
flew  two  twenty-four  minute  test  trials  with  each  display  presentation  mode,  one  with 
the  eye  movement  recorder  and  one  without.  Half  of  the  subjects  were  tested  with 
the  eye  movement  recorder  present  on  the  first  set  of  trials.  The  remaining  half  were 
tested  with  the  eye  movement  recorder  absent  on  the  first  set  of  trials.  The 
presentation  order  for  levels  of  the  display  mode  and  eye  movement  — iriables  is  shown 
in  Table  4.3-1.  As  can  be  seen,  the  levels  of  these  variables  were  counterbalanced 
over  the  eighteen  subjects. 


The  pilots  wore  a  set  of  headphones  and  a  microphone  during  all  trials.  They  were 
asked  for  verbal  system  star '2  reports  and  for  verbal  confirmation  of  the  presence  and 
meaning  of  emergency  conditions.  These  responses  were  recorded  on  the  audio 
channel  of  tlie  eye  movement  video  tape  for  subsequent  analysis.  When  the  eye 
movement  recorder  was  not  worn  by  the  pilot,  a  video  camera  recorded  an  image  of 
the  front  panel. 


Fuel  Crossfeed  —  CLOSED 

Fuel  Transfer  Switches  —  OFF 

Flaps  -  UP,  Sweep  25  degrees 

Engine  and  Fuel  Master  -  ON 

Throttles  to  MILITARY  Power  (in  detent) 

Electrical  -  LH  and  RH  Generators  ON 
Gear  —  liP 

Standby  Altimeter  Setting  -  28.92 
Program  Stores:  (left  MPD) 

Select: 

ECM  -  Standby 

BLU  —  Ripple,  Qty  2, 50  feet 

MK  82  -  Ripple,  Qty  12:  Interval  100  feet; 

nose  and  tail  fused 
AAM  —  Single 
Select  BLU 
Master  Arm  -  OFF 
HSD  -  Select  320  nm  Scale 
Master  Mode  Control  -  Select  N  AV  Mode 
Engine  Status  -  ON  (right  MPO) 

Adjust  Seat  and  Ruddt'  Pedals 

Autopilot  -  AUTOPI  LOT  ENGAGED 

Hydraulic  Control  -  T-bar  OFF 

Engage  RUN  on  Simulator  Control  Panel  When  Ready 


Figun  4.3-2.  Pn-Flfght  Checklist 


Table  4.3-1.  Sequence  of  Conditions  for  Test 
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Forty  minutes  were  reserved  in  the  daily  test  schedule  to  complete  each  practice  and 
test  tr-al.  This  included  a  sixteen-minute  period  to  reconfigure  the  simulator,  rotate 
subjects,  and  calibrate  the  eye  movement  recorder  for  a  subsequent  trial. 

Control  of  the  simulation  facility  was  turned  v/ver  to  the  test  conductor  only  after  the 
facility  had  been  initialized  by  the  laboratory  personnel.  Following  each  test  run, 
control  of  the  facility  was  returned  to  the  laboratory  personnel  to  allow  them  to  re¬ 
initialize  the  equipment  for  the  next  test  condition.  During  the  re-initialization 
period,  the  eye  movement  recorder  was  recalibrated  as  required  and  data  acquisition 
was  verified. 


4.4 


Mission  Scenarios 


The  navigation  track  for  the  simulated  mission  is  shown  in  Figure  4.4-1.  Table  4.4-1 
defines  pertinent  aircraft  conditions  and  mission  events  for  each  leg  of  the  flight  plan. 
The  mission  covered  approximately  240  nautical  miles.  During  each  trial  the  pilot 
experienced  both  Visual  Meteorological  Conditions  (VMC)  and  Instrument 
Meteorological  Conditions  (IMC),  manual  and  coupled  flight,  air-to-air  and  air-to- 
ground  weapon  deliveries,  degraded  flight  conditions,  and  airspeeds  from  500  knots  to 
Mach  1 .6. 

Leg  A 

The  mis  ion  began  with  the  aircraft  on  autopilot  holding  altitude  at  20,000  feet  and 
heading  at  330  degrees.  The  power  was  set  to  500  knots  for  cruise  to  the  Forward 
Edge  of  the  Battle  Area  (FEBA).  The  aircraft  had  just  completed  aerial  refueling  and 
was  proceeding  inbound  to  the  pre-assigned  target  area,  a  petroleum  dump  located  at  a 
railroad  switching  yard. 

Shortly  after  initiation  of  the  miction,  the  pilot  received  a  communication  from  an 
advanced  C^  aircraft  informing  him  that  two  enemy  aircraft  were  proceeding 
outbound  from  the  FEBA  on  an  intercept  course.  The  pilot  set  the  ECM  pods  to  ON, 
selected  missiles  as  the  active  weapons  option,  activated  the  master  arm  switch,  and 
selected  the  air-to-air  master  mode  in  preparation  for  the  aerial  engagement. 

Two  enemy  aircraft  appeared  at  the  top  of  the  HSD.  Shortly  after  the  missile  lock -on 
appeared  on  the  MUD,  the  pilot  launched  a  missile  and  one  of  the  enemy  aircraft  was 
destroyed.  The  second  enemy  aircraft  was  beyond  range  for  missile  lock-on  so  the 
aerial  engagement  was  terminated.  At  this  time  the  pilot  set  the  ECM  pods  to 
standby,  selected  napalm  as  the  active  weapons  option,  turned  the  master  arm  switch 
off,  and  selected  the  navigation  master  mode. 

A  "systems  status"  message  was  presented  over  the  pilot's  headset  as  a  cue  to  call  up 
the  subsystem  status  formats  by  depressing  the  appropriate  MPD  mode  select 
switches.  As  each  switch  was  depressed,  the  appropriate  subsystem  format  appeared 
on  one  of  the  MPDs.  Both  normal  and  degraded  system  status  formats  were  presented. 
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Table  4.4-1.  Flight  Plan  Events 
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Control 

Altitude 
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The  pilot  verbally  identified  each  subsystem  as  it  was  selected  and  reported  "normal" 
or  "degraded"  as  appropriate.  If  a  degraded  format  was  presented,  the  pilot  also 
provided  a  brief  verbal  description  of  the  nature  of  the  problem  and  performed  the 
appropriate  corrective  action,  if  required. 


The  aircraft  descended  from  20,000  feet  to  minimum  terrain  following  altitude 
immediately  after  the  air-to-air  engagement.  During  descent,  the  pilot  performed 
another  systems  status  check  and  received  a  message  informing  him  that  the 
petroleum  dump  had  been  destroyed,  and  directing  turn  to  bomb  the  switching  yard. 
Upon  receipt  of  this  message,  the  pilot  selected  bombs  as  the  active  weapons  option. 
As  the  aircraft  paused  through  10,000  feet,  the  pilot  selected  TF/TA  on  the  master 
mode  panel  and  the  low  altitude  displays  appeared. 

The  aircraft  encountered  VMC  shortly  after  reaching  terrain  following  altitude.  The 
total  VMC  simulation  period  lasted  approximately  three  minutes.  The  VMC  period  was 
used  to  evaluate  the  degree  of  visual  interference  between  HUD  symbology  and  a 
dynamic  color  background,  particularly  in  the  color  presentations. 


♦3 


The  aircraft  autopilot  captured  a  heading  of  040  degrees  at  the  beginning  of  Leg  C. 
The  aircraft  returned  to  IMC  and  remained  in  that  condition  for  the  rest  of  the  flight. 
Another  systems  status  check  was  performed.  The  VSD  and  Horizontal  Situation 
Display  (HSD)  formats  were  used  to  observe  the  position  of  enemy  ground  threats 
during  the  approach  to  the  target  area. 

Le&D 

The  aircraft  autopilot  captured  a  heading  of  315  degrees  at  the  beginning  of  Leg  D  and 
the  pilot  assumed  manual  control  of  the  airplane.  The  target'  area,  a  railroad 
switching  yard,  was  highlighted  on  the  HSD  as  the  aircraft  approached  the  weapon 
release  point.  All  twelve  MK  82  bombs  were  delivered  from  a  wingS-level  attitude  on 
a  single  pass  at  550  knots.  Weapon  delivery  was  cued  by  the  appearance  of  a  weapon 

release  symbol  on  the  Head-Up  Display  (HUD).  Following  weapon  release,  the  pilot 
maintained  minimum  terrain-following  altitude  for  withdrawal  from  the  target  area. 

A  single  hung  bomb  remained  following  weapon  delivery  on  selected  trials.  When  this 
occurred,  the  pilot  deactivated  the  master  arm  switch,  deselected  bombs  as  the  active 
weapons  option,  and  defused  the  hung  bomb. 

LegE 

The  aircraft  was  flown  manually  to  a  heading  of  250  degrees  at  waypoint  5  and  speed 
was  increased  to  600  knots.  The  pilot's  primary  task  during  this  leg  was  to  use  the  VSD 
and  HSD  formats  to  minimize  exposure  to  enemy  ground  threats.  Displayed  ground 
threats  included  both  prebriefed  and  pop-up  threats*  Prebriefed  threats  remained  on 
the  displays  as  long  as  they  were  within  the  selected  range.  Pop-up  threats  appeared 
suddenly  as  an  addition  to  the  prebriefed  threats.  When  a  pop-up  threat  appeared,  the 
pilot  activated  a  navigation  update  switch  that  fed  the  new  threat  data  to  the 
navigation  computer  so  that  the  flight  channel  could  be  modified  based  on  computer- 
aided  analysis  of  the  best  evasive  tactic.  Thirty  seconds  were  required  to  generate  a 
new  flight  channel  following  the  pilot's  input  to  the  navigation  system.  Therefore,  a 
control  input  to  update  the  navigation  system  had  to  be  made  at  least  30  seconds  prior 
to  intercepting  the  pop-up  threat  if  it  was  to  be  effective.  All  pop-up  threats  were 
presented  from  7  to  12  miles  away  from  Lie  aircraft.  At  an  airspeed  of  600  knots,  the 


pilot  had  a  maximum  of  12  seconds  to  detect  and  respond  to  a  pop-up  threat  that  was  7 
miles  away  and  a  maximum  of  k2  seconds  to  detect  and  respond  to  a  pop-up  threat 
that  was  12  miles  away.  Navigation-update  commands  at  other  times  in  the  mission 
were  assumed  to  be  assimilated  by  the  navigation  system,  but  did  not  actually  result  in 
alterations  to  the  flight  channel.  When  evasion  tactics  were  employed,  the  flight 
channel  was  programmed  to  reintercept  the  mission  leg  once  the  threat  had  been 
avoided. 

Leg  F 

The  autopilot  was  engaged  at  waypoint  6  after  crossing  the  FEBA  outbound.  The 
aircraft  captured  a  heading  of  190  degrees  and  the  afterburner  was  used  during  a  Mach 

1.6  dash  to  cruising  altitude  of  20,000  feet.  Power  was  reduced  to  600  knots  at  top  of 
climb  and  a  systems  status  check  was  performed. 

Le*G 

The  aircraft  autopilot  captured  a  heading  of  126  degrees  at  the  beginning  of  Leg  G. 
During  the  return  to  home  base,  the  pilot  exercised  the  fuel  range  ring  feature  of  the 
HSD.  A  final  systems  status  check  was  performed  in  Leg  G. 

*.5  Emergency  Conditions  and  System  Status  Reports 

Emergency  conditions  were  presented  on  selected  trials  in  Legs  B,  C,  D,  E,  F,  and  G. 
Warning  symbology  appeared  in  the  HUD,  the  master  caution  light  came  on  and 
selector  buttons  blinked  for  the  appropriate  status  and  advisory  displays.  When  this 
occurred,  the  pilot  selected  the  proper  displays,  verbally  identified  the  nature  of  the 
problem  and  performed  the  appropriate  corrective  action,  if  required.  Formats  for  the 
following  emergency  conditions  were  presented: 

Left  generator  failure  -  temporary* 

Left  generator  failure  -  permanent*  . 

Right  generator  failure  -  temporary* 

-  Right  generator  failure  -  permanent* 

-  Hydraulic  circuit  failure  -  IB 
Hydraulic  circit  failure  -  2A 


Fuel  boost  pump  failure 
Hung  bomb* 

Left  engine  out* 

Left  engine  overtemp* 

Left  engine  fire* 

Right  engine  out* 

Right  engine  fire* 

Some  of  the  emergency  conditions  required  the  pilot  to  take  immediate  corrective 
action.  An  asterisk  (*)  has  been  used  to  indicate  all  such  conditions  in  the  above  list. 
No  emergency,  with  the  exception  of  the  hung  bomb,  occurred  more  than  once  in  the 
same  mission  leg  over  the  six  trials  for  a  given  pilot.  The  top  half  of  Table  4.5-1 
shows  the  assignment  of  specific  failures  to  mission  legs  for  each  of  the  six  trials.  A 
dashed  line  in  the  body  of  the  table  signifies  that  no  emergency  format  was  presented 
for  a  particular  leg  on  a  particular  trial.  The  trial  numbers  in  the  left  column 
correspond  to  the  numbers  in  the  body  of  Table  4.3-1,  Sequence  of  Test  Conditions. 

System  status  reports  were  requested  during  Legs  A,  B,  C,  F,  and  G.  In  addition  to 
normal  systems  status,  verbal  reports  of  the  following  degraded  conditions  were 
required: 

-  Fuel  boost  pump  failure 

-  Closed  valve  right  wing  fuel  tank  -  temporary* 

-  Closed  valve  left  wing  fuel  tank  -  temporary* 

Right  transformer  rectifier  failure  -  permanent 

-  Left  transformer  rectifier  failure  -  permanent  ' 

-  Right  generator  failure  -  permanent 

-  Left  generator  failure  -  permanent 

-  Hydraulic  circuit  IB  -  temporary 

-  Hydraulic  circuit  2A «» temporary 

-  Hydraulic  circuit  IB  -  permanent 

-  Hydraulic  circuit  2A  -  permanent 
Hung  Bomb 

Left  engine  out 
Right  engine  out 


Two  fuel  conditions  required  the  pilot  to  take  immediate  corrective  action,  as 
indicated  by  asterisks.  The  bottom  half  of  Table  4.5-1  shows  the  assignment  of 
specific  status  formats  to  mission  legs  for  each  of  the  six  trials.  No  routine  status 
reports  were  requested  during  Legs  D  and  E  when  the  pilot  was  flying  manually.  The 
letters  F,  H,  and  E  adjacent  to  the  trial  numbers  in  the  left  column  stand  for  the  fuel, 
hydraulic,  and  electrical  subsystems,  respectively.  Status  formats  were  determined,  in 
some  cases,  by  previously  presented  emergency  conditions.  If,  for  example,  an 
emergency  format  was  presented  for  a  permanent  right  generator  failure,  all 
subsequent  electrical  status  formats  indicated  a  permanent  right  generator  failure. 
The  number  of  degraded  system  status  formats  increased  from  Leg  A  to  Leg  G  because 
each  permanent  failure  remained  on  all  subsequent  legs  after  its  initial  presentation. 
Stores  status  and  normal  fuel  range  were  added  to  the  Leg  F  status  request.  Stores 
status,  engine  status  and  extended  fuel  range  were  added  to  the  Leg  G  status  request. 

After  each  pilot  had  been  exposed  to  all  of  the  test  conditions,  a  questionnaire  and  a 
structured  interview  were  administered  to  elicit  constructive  comments  about  the 
pictorial  formats  and  obtain  comparative  judgments  about  the  usefulness  of  the  three 
display  presentation  modes.  The  debriefing  emphasized  the  identification  of  problem 
areas  and  suggestions  for  improvements  to  the  display  formats.  The  test  conductor 
was  present  during  the  debriefing  period  to  answer  any  questions  the  subjects  might 
have  and  to  ensure  that  all  questionnaire  items  were  answered  completely. 

I  ' 

1.6  Test  Data  Collection 

It  is  not  uncommon  to  obtain  ccnflicting  results  when  objective  and  subjective  data 
collection  techniques  are  used.  Differences  between  pilot  performance  and  pilot 
opinion  may  suggest  differential  sensitivity  between  the  two  measurement  techniques. 
Therefore,  both  sorts  of  data  were  collected  in  this  study. 

L6.1  Pilot  Performance  Measures 

1  light  measures  of  pilot  performance  were  recorded.  These  measures  provided 
quantitative  data  with  regard  to  the  pilots1  ability  to  use  pictorial  information  in  the 
:hree  display  presentation  modes  to  accomplish:  a)  flight  path  control,  b)  threat 
detection  and  avoidance,  c)  weapon  delivery,  d)  identification  and  resolution  of 


degraded  system  status,  and  e)  identification  and  resolution  of  emergency  conditions. 
It  was  assumed  that  deviations  from  the  flight  path  channel,  problems  in  threat 
detection  and  avoidance,  errors  in  weapon  delivery,  and  difficulty  in  the  identification 
and  resolution  of  degraded  or  emergency  conditions  would  be  greater  for  display 
presentation  modes  that  were  more  difficult  to  use. 

Percent  of  Time  Airolane  Symbol  Within  Flieht  Path  Channel 


Accuracy  of  flight  path  control  was  measured  during  Legs  D  and  E  when  the  pilots 
were  flying  manually.  The  magnitude  of  vertical  and  lateral  excursions  from  the 
center  of  the  flight  path  channel  were  recorded  at  a  rate  of  one  sample  per  second. 
Data  collection  began  when  the  pilot  deselected  the  autopilot  at  the  beginning  of  Leg 
D  and  ended  when  the  pilot  was  instructed  to  engage  the  autopilot  at  the  end  of  Leg  E. 


Response  latency  to  presentation  of  pop-up  threats  was  measured  in  Legs  C  and  E. 
Response  latency  was  defined  as  the  interval  from  the  appearance  of  the  pop-up  threat 
on  the  HSD  to  the  pilot's  control  input  to  update  the  navigation  system. 

to  Missile  Lock-on  Cue 


Response  latency  to  presentation  of  the  missile  lock-on  cue  was  measured  in  Leg  A. 
Response  latency  was  defined  as  the  interval  from  the  appearance  of  the  lock-on  cue 
on  the  HUD  to  the  pilot's  control  input  to  fire  a  missile. 

to  Bomb  Release  Cue 

Response  latency  to  presentation  of  the  bomb  release  cue  was  measured  in  Leg  D. 
Response  latency  was  defined  as  the  interval  from  the  appearance  of  the  bomb  release 
cue  on  the  HUD  to  the  pilot's  control  input  for  weapon  release. 


ion  of  Errors  in 


Response  latency  and  error  data  for  system  status  formats  was  measured  in  Legs  A,  B, 
C,  F,  and  G.  Response  latency  was  defined  as  the  interval  from  the  end  of  the  status 


request  to  beginning  of  the  response.  An  error  was  defined  as  an  incorrect  status 
report  ("normal"  when  actually  degraded  or  "degraded"  when  actually  normal)  or  a 
failure  to  perform  the  appropriate  corrective  action. 

Latency  of  Response  and  Proportion  of  Errors  in  Responding  to  Emergency  Formats 

Response  latency  and  error  data  for  emergency  formats  was  measured  in  Legs  B,  C,  D, 
E,  F,  and  G.  Response  latency  was  defined  as  the  interval  from  the  appearance  of  the 
emergency  format  to  the  beginning  of  the  pilot's  verbal  response  identifying  the  nature 
of  the  problem.  All  engine,  electrical,  and  stores  emergency  formats  required  one  or 
more  overt  motor  responses  by  the  pilot.  An  error  was  defined  as  an  inappropriate 
control  input,  ah  omission  of  a  specified  control  input,  or  as  an  incorrect  verbal 
analysis  of  the  problem. 

4.6.2  Eye  Fixations  and  Movements 

A  continuous  record  of  eye  fixations  and  eye  movements  within  the  cockpit  was 
recorded  for  all  pilots  in  each  of  the  display  presentation  modes.  Performance 
measures  included: 

Median  dwell  time  on  each  display 

-  Proportion  of  time  srant  fixating  each  display 

-  Proportion  of  fixe  .ions  on  each  display 
Fixation  rate  or.  each  display 

-  Transitional  probabilities  from  one  display  to  another 

This  data  was  recorded  on  video  tape  for  subsequent  reduction  and  analysis.  Although 
care  must  be  taken  in  interpretating  such  data,  the  pattern  cf  eye  movements,  in 
conjunction  with  other  performance  measures  such  as  response  latencies,  may  reveal 
display-format  deficiencies. 

4.6.3  Pilot  Opinion  Measures 

A  comprehensive  questionnaire  was  administered  after  all  simulation  test  trials  were 
completed.  The  questionnaire  was  composed  of  three  parts.  The  first  part  was 
designed  to  examine  the  pilots'  opinions  regarding  the  relative  usefulness  of  the  three 
presentation  modes  and  to.  evaluate  the  extent  to  which  the  individual  formats  were 


integrated  into  a  coherent,  and  comprehensive  information  presentation  system.  The 
second  part  was  designed  to  examine  the  pilots'  general  level  of  acceptance  of 
pictorial  displays  and  to  compare  their  relative  usefulness  with  conventional  aircraft 
displays.  The  third,  and  final,  part  of  the  questionnaire  was  designed  to  obtain  critical 
comments  regarding  display  dynamics  and  specific  features  withu,  display  formats. 
The  questionnaire  addressed  differences  between  high  and  low  altitude  versions  of  the 
primary  flight  and  navigation  displays.  Topics  receiving  special  attention  included: 

Flight  path  channel  on  the  HUD  during  IMC  versus  VMC 

-  Terrain  presentation  on  the  HUD  during  IMC  versus  VMC 
Terrain  and  threat  presentation  on  the  VSD 
Perspective  flight  path  on  the  VSD 

-  Terrain  and  threat  presentation  on  the  HSD 

-  Target  expanded  inset,  and  fuel  range  ring  options  on  the  HSD 

-  Stores  formats 
Engine  formats 

-  Air-to-air  format  on  the  HSD 

-  All  system  status  formats 

All  emergency  condition  formats 
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RESULTS  -  5 ASIC  STUDY 


5.1  Performance  Data  and  Multivariate  Analysis  of  Variance 

An  on-line  data  acquisition  system  recorded  the  time  of  selected  discrete  events. 
Output  from  the  on-line  system  contained  the  computer  clock  time  for  the  following 
events: 

When  the  pilot  deselected  autopilot  at  the  beginning  of  leg  D 
When  the  pilot  selected  autopilot  at  the  end  of  leg  E 
When  a  pop-up  threat  was  presented 
When  the  pilot  activated  the  navigation  update  switch 
When  the  missile  lock-on  cue  was  presented  on  the  HUD 
When  the  bomb  release  cue  was  presented  on  the  HUD 
When  the  pilot  released  a  missile 
When  the  pilot  released  bombs 

When  the  pilot  activated  the  air-to-air  master  mode  switch 
When  the  pilot  activated  the  MPD  mode  select  switches 
When  an  emergency  format  was  presented 
When  the  right  or  left  generator  switches  were  turned  on  or  off 
When  the  right  or  left  throttle  was  moved  to  the  idle  or  off  position 
When  the  pilot  activated  the  engine  restart  switch 
When  the  pilot  received  each  pre-recorded  message 

In  addition,  the  output  contained  RMS  errors  for  lateral  and  vertical  deviations  from 
the  flight  channel  as  well  as  total  time  and  percent  of  time  that  the  airplane  symbol 
was  within  the  flight  path  channel  during  manual  flight.  This  information  was 
generated  at  the  end  of  each  trial. 

Error  data  and  response  latency  data  for  pilot's  system  status  reports  and  responses  to 
emergency  conditions  were  extracted  from  the  audio  and  the  video  channels  of  the 
video  tapes.  The  error  data  were  obtained  by  monitoring  the  pilot's  verbal  and  motor 
responses  and  recording  whether  they  were  correct  or  incdrect.  Response  latencies 
for  system  status  reports  were  obtained  by  searching  for  successive  requests  for 


system  status  information  on  the  audio  channel.  For  each  such  request,  the  digital 
clock  time  at  the  beginning  of  the  pilot's  response  was  recorded.  The  difference 
between  the  message  presentation  time  and  the  beginning  of  the  pilot's  response 
defined  the  latency  of  response  measure. 

Response  latencies  for  emergency  conditions  were  obtained  by  locating  successive 
emergencies  on  the  video  channel.  For  each  emergency  condition,  the  digital  clock 
time  was  recorded  at  the  beginning  of  the  pilot's  verbal  response.  The  difference 
between  the  emergency  onset  and  the  beginning  of  the  pilot's  response  defined  the 
latency  measure  for  emergency  conditions. 

A  preliminary  multivariate  analysis  of  variance  (MANOVA)  was  performed  to 
determine  if  the  presence  of  eye  movement  recorder  had  an  appreciable  effect  on  the 
performance  measures.  The  independent  variables  in  this  analysis  were  eye  mark 
recorder  with  two  levels  (present  or  absent)  and  display  presentation  mode  with  three 
levels  (monochrome,  color  line  or  color  fill).  The  dependent  measures  included: 
percent  of  time  in  150  by  300  foot  flight  channel,  root  mean  square  vertical  deviation 
'  from  flight  channel,  root  mean  square  lateral  deviation  from  flight  channel,  latency  of 
response  to  pop-up  threats,  latency  of  response  to  missile  lock-on  cue,  latency  of 
response  to  bomb  release  cue,  and  latency  of  response  to  system  status  requests.  Jhe 
MANOVA  summary  table  for  this  analysis  is  shown  in  Table  5.1-1.  No  significant  main 
or  interaction  effects  involving  the  eye  movement  recoder  variable  were  found. 

Since  the  eye  movement  recorder  did  not  differentially  affect  the  other  performance 
measures,  these  measures  were  averaged  over  the  eye  mark  and  no  eye  mark 
conditions  and  a  second,  one-way  MANOVA  was  performed.  The  independent  variable 
in  this  analysis  was  display  presentation  mode  with  three  levels  (monochrome,  color 
line,  or  Color  fill).  The  same  seven  dependent  measures  were  used  in  this  analysis. 
The  MANOVA  summary  table  for  this  analysis  is  shown  in  Table  5.1-2.  Results  of  the 
MANOVA  indicated  no  significant  difference  among  the  three  display  presentation 
modes  for  the  set  of  seven  performance  measures. 

Table  5.1-3  shows  the  mean  and  standard  deviation  for  each  performance  measure  in 
the  three  display  presentation  coriditions.  As  can  be  seen,  three  of  the  seven 
performance,  measures  (RMS  verticr  i  deviation,  RMS  lateral  deviation,  and  latency  to 


Table  5. 1- 1.  Two-Factor  Repeated  Measures  MANO  VA 


MANOVA  for  pictorial  format  diipiay  (valuation 
Effact  . .  eyemark 

Multivariata  tests  of  significance  (£  •  1,  M  •  2%.  N  *  4%) 


Test  name 

Value 

Approximate  F 

Hypothesis  DF 

Error  OF 

Significance  of  F 

Pillais 

.57768 

2.14870 

7.00 

11.00 

.  .124 

Hoteilingi 

1.36735 

2.14870 

7.00 

11.00 

.124 

Wilks 

.42241 

2.14870 

7.00 

11.00 

.124 

Roys 

.57758 

MANOVA  for  pictorial  format  display  evaluation 
Effact  .  .  display 

Multivariate  tests  of  significance  (S  -  1.  M  »  6,  N  *  1 ) 


Test  name 

Value 

Approximate  F 

Hypothesis  DF 

Error  OF 

Significance  of  F 

Pillais 

.76368 

HuLiLjf'MHK 

14.00 

4.00 

.598 

Hoteilings 

3.23157 

14.00 

4.00 

.598 

Wilks 

.23632 

.92331 

14.00 

4.00 

.598 

Roys 

.76368 

■ 

MANOVA  for  pictorial  format  display  evaluation 

Effact  . .  eyemark  by  display 

Multivariate  tests  of  significance  (S  -  1.  M  »  6.  N  -  1) 


Test  name 

Value 

Approximate  F 

Hypothesis  DF 

Error  DF 

Significance  of  F 

Pillais 

.87741 

2.04489 

14.00 

4.00 

.256 

Hoteilings 

7.15711 

2.04489 

14.00 

4.00 

.256 

Wilks 

.12258 

2.04489 

14.00 

4.00 

.256 

Roys 

,877s  1  ’ 

Table  5.1 -2.  One-Factor  Repeated  Measures  MANOVA 

MANOVA  for  pictorial  format  presentation  mode 
Effact  ..  display 


Multivariata  tests  of  significance  (S  -  1.  M  •  6,  N  «  1 ) 


Test  name 

Value 

Approximate  F 

Hypothesis  DF  ' 

Error  DF 

Fillais 

.70337 

.67748 

14.00 

4.00 

.738 

Hoteilings 

3.37118 

.67748 

14.00 

4.00 

.738 

Wilks 

.29663 

.8774B 

14.00 

4.00 

.738 

Roys 

.70337 

BBSS 


Table  5. 1-3.  Mean  and  Standard  Deviation  for  Seven  Performance  Measures 


Performance  measure 

. . 1 

Monochrome  ] 

Color  line 

Color  fill 

lx 

S0 

x| 

SO 

X 

SO 

%  time  in  150  x  300  ft  channel 

76.0 

i 

7.5 

74.4 

9.4 

77.6 

10.3 

RMS  vertical  deviation 

141.3 

75.2 

132.7 

39.8 

121.8 

65.9 

RMS  lateral  deviation 

279.3 

157.3 

257.3 

135.5 

.  249.5 

143.2 

La'ancy  to  popup  threats 

17.7 

6.5 

15.5 

6.0 

17.6 

6.8 

Latency  to  miatile  lock -on 

3.1 

3.4 

2.1 

1.2 

3.0 

3.7 

Latency  to  bomb  release  cue 

2.7 

2.0 

2.6 

2.0 

3.6 

2.6 

Latency  to  system  status  request 

13.0  . 

3.1 

12.6 

1.4 

12.4 

1.5 

. 

pop-up  threats)  showed  a  consistent  improvement  from  the  monochrome  condition  to 
the  color  fill  condition.  In  all  cases,  however,  the  variability  of  scores  within  each 
condition  was  much  greater  than  the  differences  ir.  mean  performance  between  the 
conditions.  Much  of  this  variability  was  caused  by  the  high  task  demands  that  were 
intentionally  built  into  the  simulation.  For  example,  on  selected  trials,  a  pop-up 
threat  was  presented  shortly  after  releasing  bombs.  In  addition,  a  hung  bomb  condition 
occurred  following  bomb  release  on  selected  trials.  A  problem  with  the  bomb,  release 
task  could  precipitate  subsequent  problems  in  dealing  with  the  hung  bomb  or  in 
identification  of  the  pop-up  threat.  Thus,  a  pure  test  of  the  three  display  presentation 
conditions  for  each  performance  measure  was  not  possible  because  of  the 
interdependence  of  task  demands.  While  the  simulation  scenario  introduced  a 
considerable  amount  of  error  variability  into  the  data,  it  also  enabled  the  test  subjects 
to  evaluate  a  large  number  of  display  concepts  in  a  realistic  mission-oriented 
environment.  The  demonstration  of  these  new  pictorial  display  concepts  was 
considered  tn  important  aspect  of  this  study. 
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5.2 


Eye  Mark  Recorder  Data 


A  frame-by-frame  analysis  of  the  eye  movement  data  during  manual  flight  was 
conducted  to  transform  each  video  record  into  a  digital  format.  A  video  control  unit 
with  single  frame  advance  capability  was  used  to  obtain  the  start  time  for  each  eye 
fixation.  The  following  procedure  was  used  to  complete  the  manual  translation  effort. 
The  cockpit  was  divided  into  seven  zones  for  data  reduction  purposes.  These  zones 
were  identified  as  follows: 


Zone  Display 


1 

2 

3 

* 

3 

6 

7 


Head-Up  Display 

Vertical  Situation  Display 

Horizontal  Situation  Display 

Left  Multipurpose  Display 

Right  Multipurpose  Display 

Other  (caret  visible  but  not  on  one  of  the  above) 

Unknown  (caret  not  visible) 


The  data  were  recorded  using  a  format  that  included  the  display  number  and  the  time 
and  frame  number  at  which  the  eye  movement  caret  first  appeared  on  a  display  for 
each  fixation.  For  example,  the  code  03013*23  denotes  a  fixation  on  display  3 
(Horizontal  Situation  Display)  beginning  at  1  minute,  3*  seconds  and  23  frames  into  the 
simulation. 


Approximately  30  hours  were  required  to  translate  1  hour  of  video  data  into  a  digital 
format.  The  video  records  of  eye  movements  and  eye  fixations  occur ing  in  Legs  D  and 
E  were  analyzed  for  all  pilots  in  each  of  the  three  presentation  modes.  These  legs 
were  chosen  for  analysis  because  of  the  high  workload  imposed  by  manual  flight, 
weapon  delivery  and  threat  avoidance  tasks. 

The  translated  eye  data  was  processed  through  a  screening  program  that  flagged 
specific  data  input  errors.  The  raw  data  were  edited  to  remove  all  identified  errors 
prior  to  data  analysis.  Then,  a  second  computer  program  was  used  to  generate 
descriptive  statistics  for  the  eye  movement  data.  Table  3.2-1  shows  mean 
performance  for  the  eighteen  subjects  on  four  eye-movement  variables:  median  dwell 

time  in  seconds,  percent  of  flight  time  spent  fixating  on  a  display,  percent  of  total 

\  .  ■  ’  *  - 


Table  5.2-1.  Eye  Movement  Date  for  Display  by  Presentation  Mode  Combinations 


Vmbb 

MonochroRW 

Color  Ihm 

Color  fill 

n 

VSD 

HSO 

MPOI 

MPD2 

HUO 

vso 

HSO 

MPOI 

MP02 

HUD 

vso 

HSO 

MPOI 

MPD2 

IMaMIM 

34 

B 

D 

06 

B 

33 

0.7 

D 

as 

06 

31 

0.8 

0.8 

0.4 

0.3 

\o«  tool  o~» 

H 

H 

22 

08 

B 

83  0 

39 

16 

1.0 

OS 

851 

3.5 

.  1.4 

07 

02 

»«#  M  fuunom 

421 

14.7 

76 

11 

14 

430 

148 

87 

3.5 

VS 

43.7 

128 

53 

3.1 

1.5 

fimoorr  nrn/mm 

SI 

2.7 

13 

D 

02 

82 

D 

D 

06 

D 

0 

23 

D 

D 

0.2 

fixations  devoted  to  a  display,  and  fixation  rate.  As  can  be  seen,  eye  movements  and 
fixations  were  dominated  by  the  HUD.  Over  SO  percent  of  the  manual  flight  time  was 
spent  fixating  the  HUD.  Median  dwell  times  were  much  longer  for  the  HUD  and 
fixation  rates  were  much  higher  for  the  HUD.  Pilot  eye  movements  and  fixations  did 
not  vary  as  a  function  of  display  presentation  mode.  Indeed,  a  display  by  display 
comparison  between  presentation  modes  reveals  nearly  identical  values  for  all  of  the 
eye  movement  variables. 

The  remainder  of  the  eye  movement  analysis  was  done  on  data  from  pilot  10,  mission 
2,  color  fill.  Table  5.2-2  summarizes  relevant  parameters  for  that  flight  segment,  and 
each  display.  By  inspection  these  data  are  sirnilar  to  those  of  Table  5.2-1  which  were 
averaged  across  pilots.  Number  and  percent  of  fixations,  total  and  percent  of  dwell 
time  and  length  of  fixation  are  all  considerably  greater  for  the  HUD. 

Table  5.2-3  shows  the  probability  of  transition  from  each  display  represented  by  the 
row?  of  the  matrix  to  each  display  represented  by  the  columns.  Examination  of 
column  one  clearly  shows  that  the  majority  of  transitions  from  all  other  displays  was 
to  the  HUD.  The  transition  probability  from  the  HUD  was  highest  when  going  to  the 
VSD.  It  should  be  noted  that  the  probability  of  transition  from  the  HUD  to  the  OTHER 
and  UNKNOWN  categories  was  higher  than  to  all  other  displays  except  the  VSD. 
Because  of  this,  the  percent  of  total  time  and  the  percent  of  total  fixations  for  these 
two  categories  were  typically  higher  than  for  the  HSD  and  the  two  MPDs.  The 
relatively  high  percentage  pf  time  for  the  OTHER  and  UNKNOWN  categories  resulted 
from  the  unique  interaction  between  the  cockpit  geometry  and  the  NAC  Model  IV  eye 
mark  recorder.  Considerable  head  rotation  was  required,  when  wearing  the  eye  mark 
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Table  5.2-2.  Summary  of  Eye  Movement  Date  One  Flight 


Table  5.2-3.  Eye  Movement  Transition  Probability  Matrix 


N.  To 

From 

HUO 

VSO 

HSD 

Uft 

MPD 

Right 

MPD 

Other 

Unknown 

HUO 

- 

.304 

.030 

.121 

.030 

.242 

.212 

.  VSD 

.929 

.  - 

0 

0 

0 

.071 

0 

HSD 

.750 

.250 

- 

0 

0 

0 

0 

Uft  MPO 

1.000 

0 

0 

- 

0 

0 

0 

RHhtMPO 

1.000 

0 

0 

0 

- 

0 

0 

Other  . 

.800 

.111 

0 

0 

0 

- 

0 

Unknown 

.571 

0 

.429 

0 

0 

0 

- 

mot  10,  MWon  2.  cotor  fiM 


recorder,  for  the  pilot  to  monitor  the  vertically  oriented  HUD,  VSD  and  HSD  displays. 
It  was  possible,  however,  for  the  pilots  to  reduce  some  of  this  head  movement  by 
deflecting  their  dyes  downward.  When  this  happened  the  eye  mark  caret  would  leave 
the  half  silvered  mirror  and  disappear  from  the  video  image.  Despite  considerable 
effort  to  acquaint  the  pilots  with  this  problem,  they  all  tended  to  revert  to  the  eye 
deflection  habit.  Over  all  subjects  and  conditions,  the  percent  of  total  time  attributed 
to  the  OTHER  and  UNKNOWN  categories  ranged  from  1.7%  to  26.3%  with  a  mean  of 


9.3%.  Over  all  subjects  and  conditions,  the  percent  ol  total  fixations  on  the  OTHER 
and  UNKNOWN  categories  ranged  from  5.2%  to  50.6%  with  a  mean  of  31.4%. 

Figure  5.2-1  shows  a  frequency  distribution  of  dwell  times  for  the  HUD.  As  can  be 
seen,  there  are  a  number  of  very  long  fixations.  The  very  long  HUD  fixations  in 
conjunction  with  the  restricted  scan  pattern  suggests  that  the  manual  flight  control 
task  with  the  HUD  display  was  very  difficult.  Another  reason  for  pilot  concentration 
on  the  HUD  is  probably  that  the  HUD  format  contained  most  of  the  information  the 
pilot  needed,  so  fewer  excursions  from  that  display  were  required. 


DWELL  TtME  (mc) 


Figurt  5.2- 1.  HUD  Ey*  Dwtfl  Tim *  Fnqutney  Distribution 


53  System  Status  Reports,  Emergency  Condition  Responses,  and  Pop-up 

Responses 

Across  pilots  and  conditions,  2160  individual  status  report  items  were  requested.  Of 
these,  only  44  were  omitted  or  incorrectly  reported.  Table  5.3-1  shows  the 
distribution  by  reports  and  conditions.  Eleven  of  these  were  in  monochrome  display 
mode,  20  in  color  line  and  13  in  color  fill.  Nineteen  were  committed  by  pilots  when 
wearing  the  eye  mark  recorder  and  25  with  no  eye  mark  recorder.  By  Chi-square  test, 
neither  of  these  effects  were  significant. 

On  the  other  hand,  two  conditions  were  more  frequently  reported  incorrectly  than  the 
others.  The  "fuel  valve  closed"  report  was  omitted  or  was  incorrect  21  times 
(p  <  .00 1 ).  Fuel  range  was  omitted  or  incorrect  10  times  (p  <.01). 

Across  pilots  and  conditions,  there  were  378  programmed  emergency  conditions.  Of 
these,  65  responses  or  corrective  actions  were  omitted  or  improper.  Table  5.3-2  gives 
the  distribution  by  system  and  condition.  Twenty  of  these  were  in  monochrome  display 
mode,  21  in  color  line  and  24  in  color  fill.  Thirty-seven  were  commited  by  pilots  when 
wearing  the  eye  mark  recorder  and  28  when  not  wearing  the  eye  mark  recorder. 
Again,  neither  the  display  mode  nor  eye  mark  effects  were  significant.  The  problem- 
type  effect  was  significant.  Compared  with  other  response  errors,  there  were' 
significantly  more  (p  <  .01)  missed  hung  bomb  reports  (25  of  54  occasions)  and 
significantly  more  (p<. 001)  missed  engine  overtemperature  reports  (11  of  18 
occasions). 

On  each  flight,  there  were  two  pop-up  threats.  These  were  shown  as  unique  shapes  on 
the  VSD  and  HSD.  As  discussed  earlier,  appropriate  pilot  response  was  to  press  the 
Navigation  Update  button.  If  this  was  done  soon  enough,  an  evasion  course  would  be 
calculated  and  displayed.  If  not,  the  flight  path  continued  on  through  the  threat 
volume.  Despite  that  motivation,  a  significant  number  of  pop-up  threats  were  missed 
as  shown  in  Table  5.3-3.  By  Chi-square  test,  the  difference  between  eye-mark  and  no 
eye-mark  was  significant  (p  <  .02).  The  presentation  mode  effect  was  only  suggestive. 

5.4  Opinion  Data 

The  formal  questionnaires  and  the  list  cf  questions  for  open-ended  responses  are  given 
in  Appendix  A. 


Table  5.3 - 1.  Errors  or  Omissions  in  System  Status  Reports 


Systems  status 

Monochroma 

Color  line 

Color  fill 

Total 

Total 

reports 

wrm 

No  EM 

EM 

No  EM 

EM 

No  EM 

errors 

Fud— normal 

0 

0 

1 

0 

0 

0 

i 

468 

Fuel— boost  pomp 

1 

0 

1 

0 

0 

1 

2 

18 

Fuel— valve  closed 

2 

m 

8 

1 

1  1 

21 

54 

Hydraulic-all 

1 

0 

i 

0 

0 

1 

540 

Electrical-all 

1 . 1 

0 

0 

0 

0 

540 

Stores— all 

1 

0 

1 

0 

0 

■ 

2 

216 

Engines— all 

1 

1 

0 

0 

0 

i 

3 

108 

Fuel  range 

0 

2 

2 

3 

3 

0 

10 

216 

Total 

6 

5 

9 

11 

4' 

9 

44 

2,160 

‘Eye  mark  recorder 


Table  5.3-2.  Errors  or  Omitted  Responses  to  Programmed  Emergencies 


Programmad  emergency 


Generator-temporary 

Generator-permanent 

Hydraulic 

Hung  bomb 

Fuel 

Engine— out 
Engine— fire 
Engine  omrtemp 

Total 


*Eya  mark  racordar 


Table  5.3-3.  Missed  Pop-Up  Threats 


Monochrome 

Color  line 

Color  fill 

Eye  mark 

38.8% 

30.5% 

27.7% 

No  eye  mark 

22.2% 

16.6% 

8.3% 

36  threats  par  cell 


In  one  part  of  the  questionnaire,  pilots  were  asked  to  use  a  score  of  100  for  color  line 
as  a  standard  and  rate  monochrome  and  color  fill  for  usability  against  a  number  of 
detailed  questions.  This  is  the  method  of  magnitude  estimation  described  in  Reference 
3.  The  Friedman  two-way  analysis  of  variance  by  ranks  (Reference  4)  was  applied  to 
the  responses  to  each  question.  The  mean  ratings  are  profiled  in  Figure  5.4-1.  The 
results  indicate  a  clear  preference  for  color  iine  over  monochrome,  and  color  fill  over 
color  line,  except  for  the  HUD.  Of  the  four  questions  on  the  HUD,  pilots  preferred 
color  line  in  the  two  VMC  cases,  though  the  effects  were  not  statistically  significant. 

In  the  second  part  of  the  questionnaire,  pilots  rated  the  pictorial  formats  used  here 
against  the  display  of  comparable  information  in  "conventional  aircraft."  The  results 
of  these  ratings  are  profiled  in  Figure  5.4-2.  In'  general,  the  pictorial  formats  were 
well  accepted.  Median  response  was  at  least  "moderately  easier"  for  all  but  the  HUD, 
engine  status  and  hydraulic  status  formats. 

The  general  trends  of  pilot  opinion  elicited  by  the  formal  questionnaire  were  detailed 
in  responses  to  a  set  of  open-ended  questions.  Responses  to  those  questions  reinforced 
the  other  questionnaire  responses.  Except  for  the  HUD,  color  fill  was  perf erred,  with 
color  line  next  and  monochrome  last.  Except  for  the  HUD  under  VMC  conditions,  the 
pictorial  formats  were  generally  well  accepted. 

The  formal  questionnaire  with  average  responses,  and  a  synopsis  of  responses  to  the 
open-ended  questions,  are  given  in  Appendices  A  and  B,  respectively. 


With  color  line  set  at  100,  the  usability  ratings  for  color  fill  and  monochrome  are: 
Relative  Usability  Ratings 


Worse 

0 


Better 

50  ICO  150  200 

I  I  I  I 


HUD  pathway  in  VMC 
HUD  pathway  in  IMC 
HUD  terrain  in  VMC 
HUD  terrain  in  IMC 
VSD  terrain  and  threats 
VSD  ribbon 
HSD  terrain  and  threats 
HSD  air-to-air 
HSD  pop-up  threat 
Stores  quantity 
Stores  status 
Engine  status  normal 
Electrical  status  normal 
Electrical  status  degraded 
Hydraulic  status  normal 
Hydraulic  status  degraded 
Fuel  status  normal 
Fuel  status  degraded 
Engine  advisory 
Electrical  advisory 
Hydraulic  advisory 

Crosschecking  flight  path  high  altitude 
Crosschecking  flight  path  low  altitude 
Crosschecking  threats 
Crosschecking  terrain 


Figun  5.4-1.  Profiles  of  Seated  Comparisons  Among  Presentation  Modes 


INTRODUCTION  -  THREAT  WARNING  STUDY 


6.0 

This  section  and  the  next  four  report  the  threat  warning  study  of  the  Pictorial  Display 
Formats  Evaluation  program.  This  second  study  was  an  extension  of  the  effort  carried 
out  in  the  first  study.  Display  formats  used  earlier  were  enhanced  by  adding  threat 
information,  while  retaining  as  much  of  the  supporting  software  and  hardware  as 
possible.  The  display  formats  of  primary  interest  in  the  threat  warning  study  were 
those  which  depicted  ,  threat-related  information.  These  included  the  low  altitude 
mode  of  the  vertical  situation  display,  which  gave  a  perspective  view  of  the  airplane 
with  surrounding  terrain  and  threat  information,  from  a  viewpoint  behind  and  above 
the  aircraft.  The  low  altitude  mode  of  the  horizontal  situation  display  (HSD)  gave 
much  of  the  same  information  but  in  a  map-like  plan  view.  The  air-to-air  mode  of  the 
HSD  provided  information  about  air-to-air  threats.  Finally,  in  the  threat  warning 
study,  the  HUD  provided  threat  alert  cues. 

The  threat  warning  study  had  three  primary  objectives.  The  first  was  to  develop  a 
candidate  concept  for  pictorial  threat  displays.  The  candidate  display  formats  were 
applicable  to  both  air-to-air  and  air-to-ground  situations,  and  were  compatible  with 
advanced  threat  warning  systems.  The  second  objective  was  to  evaluate  the  usability 
and  acceptability  of  the  candidate  threat  warning  formats.  The  third  objective  was  to 
determine  if  the  degree  of  usability  and  acceptability  of  the  candidate  display  concept 
was  a  function  of  two  basic  display  presentation  modes:  color  and  monochromatic. 


7.0  TESt  EQUIPMENT  AND  FACILITIES  -  THREAT  WARNING  STUDY 

The  simulation  hardware  configuration  was  retained  from  the  first  study  to  the  second 
with  very  few  exceptions.  First,  the  eye-mark  recorder  and  the  71  mm  motion  picture 
projector  were  not  used.  Second,  a  push-button  switch  on  the  flight  control  stick  and 
an  indicator  labeled  ''EVADE''  in  the  bank  of  switches  between  the  VSD  and  the  HSD 
were  activated.  This  switch  was  used  by  the  pilot  to  indicate  that  a  short-term 
maneuver  should  be  flown  to  evade  an  incoming  missile.  With  these  two  exceptions, 
the  configuration  described  in  Section  2  applies. 
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TEST  FORMATS  -  THREA  r  WARNING  STUDY 


SU> 

As  in  the  basic  study,  primary  flight  and  threat  information  was  presented  on  three 
displays:  a  Horizontal  Situation  Display  (HSD)  and  Vertical  Situation  Display  (VSD) 
located  in  the  center  of  the  front  instrument  pane  ;  and  a  Head  Up  Display  (HUD) 
located  just  above  the  VSD.  HSD  and  VSD  information  was  presented  on  two  eight- 
inch  (diagonal)  Cathode  Ray  Tube  (CRT)  displays.  HUD  symbology  was  projected  onto 
a  color-capable  combiner  designed  especially  for  this  simulation.  The  system  status 
displays  from  the  basic  study  were  available  and  some  were  used  in  the  threat  warning 
study. 

Each  of  the  pictorial  formats  on  these  displays  was  presented  in  two  versions,  or 
presentation  types:  color  and  monochrome.  In  the  monochrome  formats,  all 
symbology  was  composed  of  black  and  white  lines  and  suri.  :es.  The  color  formats 
consisted  of  colored  outlines  and  surfaces.  Each  simulator  flight  utilized  one  of  the 
two  versions. 

Several  types  of  threat-related  information  were  represented  in  the  primary  flight 
displays:  threat  location,  threat  type,  threat  lethality,  threat  mode  (inactive,  search, 
track  or  launch),  and  countermeasures  effectiveness.  Not  all  information  was 
presented  in  all  three  displays,  but  wherever  possible  the  coding  of  threat  information 
was  similar  across  displays  and  for  both  surface-to-air  (S/A)  and  air-to-air  (A/ A) 
threats* 

S.1  Primary  Flight  Displays 

The  three  primary  flight  displays  presented  flight  information  from  three  different 
viewpoints,  updated  in  real  time.  The  Head-Up  Display  (HUD)  presented  a  pilot's  eye 
view  of  the  outside  world,  with  terrain  outlines,  threat  alert  information,  "own" 
airplane  symbol  and  the  desired  flight  path  superimposed  on  the  visual  scene  (Figure 
S.l-la).  The  HUD  fieid  of  view  was  23°  (horizontal)  by  17°  (vertical). 

The  Vertical  Situation  Display  (VSD)  displayed  much  of  the  same  information,  but  from 
a  viewpoint  6,000  ft.  behind  and  1,000  feet  above  the  aircraft,  and  with  a  wider  field 
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of  view  (90°  itorizontai).  From  this  viewpoint,  one  looked  down  and  forward  at  the 
current  aircraft  position.  In  addition  to  terrain  outlines,  airplane  position  and  the 
desired  flight  path,  threat,  and  some  terrain  altitude  information  were  presented 
(Figure  8.1-lb).  The  VSD  was  truly  a  situation  display,  using  a  remote  viewpoint  to 
include  one's  own  aircraft.  Aircraft  maneuvers  initiated  by  the  pilot  were  reflected  by 
the  movement  of  the  ribbon  and  orientation  of  the  aircraft  symbol  in  the  display;  the 
remainder  of  the  display  was  airplane  stabilized. 


The  Horizontal  Situation  Display  (HSD)  gave  a  view  looking  straight  down  from  a  point 
directly  above  the  airplane.  It  showed  a  plan  view  of  the  terrain  (including  some 
terrain  altitude  information)  and  threat  information,  along  with  the  flight  path  and 
"own"  airplane  position  (Figure  8.1-lc). 


It  is  important  to  note  that  for  each  of  the  three  primary  displays,  the  airplane  symbol 
showed  the  airplane  in  the  same  position  relative  to  the  terrain  (the  plane's  current 
position);  only  the  viewpoints  differed.  Figures  8.1-la,  b,  and  c  show  the  viewpoints 
represented  by  the  HUD,  VSD  and  HSD  for  a  given  airplane  position. 


The  VSD  and  HSD  had  two  modes:  navigation  (NAV),  and  Terrain  Following/Terrain 
Avoidance  (TF/TA).  In  addition,  the  HSD  had  an  Air  to  Air  (A/A)  Mode.  These  modes 
were  selected  using  the  master  mode  switches  located  between  the  VSD  and  HSD. 
Pressing  the  NAV  or  TF/‘  'A  mode  switches  put  the  selected  format  on  both  the  VSD 
and  HSD.  The  A/A  mode  switch  w«.s  "press-on,  press-off"  and  affected  the  HSD  only. 
Pressing  it  on  caused  the  \/A  HSD  mode  (160  mile  range)  to  be  displayed.  Pressing  it 
off  returned  the  HSD  to  die  previously  selected  mode  and  scale.  The  range  selection 
switches  for  the  HSD  operated  in  all  three  modes. 


S.1.1 


Head-Up  Display  (HUD) 


An  example  of  the  Head  Up  Display  (HUD)  TF/TA  format  is  shown  in  Figure  8.1-2. 
Outlines  of  terrain  features  (mountains)  and  the  horizon  were  projected  on  the  HUD. 
In  addition,  a  segmented  flight  channel  (pathway  in  the  sky)  defined  the  desired  flight 
path.  The  entry  gate  to  the  flight  channel  was  300  feet  wide  and  150  feet  high.  The 
flight  channel  started  4,0  )0  feet  in  front  of  the  plane  and  extended  6,000  feet.  Lines 
on  the  floor  of  the  "pathway"  represented  1000  foot  distances  along  the  desired  flight 


path.  Ii  the  desired  flight  path  Was  maintained  accurately,  the  aircraft  symbol  was 
positioned  in  the  middle  of  the  "pathway"  entry  gate,  and  its  "wings"  were  lined  up 
evenly  with  the  lines  that  extended  outward  from  the  middle  of  the  sides  of  the  flight 
channel. 

If  the  desired  flight  path  was  not  maintained,  the  pathway  moved  relative  to  the 
aircraft  symbol,  which  remained  fixed.  The  entrance  of  the  flight  channel  in  Figure 
S.l-2  is  displaced  slightly  below  the  airp.’ine  symbol.  This  indicates  that  the  pilot 
should  descend  s*j  *Kt'y  to  regain  the  desired  flight  path.  If  the  aircraft  deviated  far 
enough  from  tie  r  iired  course,  the  pathway  moved  out  of  the  theoretical  field  of 
view  of  the  HUD.  It  did  not  disappear  from  the  HUD,  however,  instead  it  was  pegged 
to  the  side  of  the  HUD  in  the  direction  of  its  current  position.  When  the  pathway  was 
pegged  to  the  side,  a  transitional  flight  director  symbol  (an  inverted  "T")  appeared 
(shown  in  Figure  8.1-3)  to  aid  in  regaining  the  correct  course.  The  transitional  flight 
director  disappeared  once  the  aircraft  was  within  five  degrees  of  the  entry  gate  of  the 
pathway. 

A  line  which  extended  from  the  end  of  the  pathway  indicated  the  desired  flight  path 
past  the  6,000  foot  pathway  limit.  Other  symbology  included  square  waypoint  "flags" 
and  triangular  target  "flags"  which  indicated  the  locations  of  these  points  relative  to 
the  desired  flight  path.  Boxed  readouts  of  airspeed  and  altitude  appeared  to  the  left 
and  right  of  the  aircraft  symbol;  heading  was  shown  at  the  top  center. 

Threat  alert  information  (detailed  in  Figure  8.1*3)  was  also  provided  on  the  HUD.  An 
airplane-  or  a  missile-shaped  "threat  alert  symbol"  represented  air-to-air  or  surface- 
to-air  threats,  respectively.,  The  appropriate  Symbol  appeared  for  six  seconds  when  a 
threat  became  active  (went  info  search  mode).  When  a  threat  went  into  track  mode, 
the  same  symbol  flashed  for  six  second'-  In  launch  mode,  a  vector  that  gave  the 
relative  azimuth  of  the  missile  or  tf  eat  site  appeared,  along  with  the  flashing  threat 
alert  symbol.  In  addition,  a  boxed  readout  of  "time  to  impact"  appeared  above  the 
threat  alert  symbol. 

The  navigation  (NAV)  HUD  format  was  identical  to  the  low  altitude  format,  except 
that  terrain  outlines  were  reduced  to  a  single  artificial  horizon. 


AIR  TO  AIR  \ 

THREAT  ALERT  SYMBOL 
(FLASHED  IN  TRACK  AND 
LAUNCH  MODES) 


TRANSITIONAL 
FLIGHT  DIRECTOR 


■THREAT 
AZIMUTH 
VECTOR  - 
LAUNCH  MODE 


-TIME  TO 
MISSILE 
IMPACT 


Information 

Level* 

Coding  in  Threat  Warning  Study 

Threat  Alart 

Search -mode 

Air-to-air  -  a>rplana  symbol  |  g 

Surface-to-air  -  missile  symbol  J 

'« 

"Track  Mode"  threat 

Air-to-air  -  flashing  airplane  symbol  lg 
Surface-to-air  —  flashing  missile  symbol  1 

"Launch  Mode"  threat 

Same  as  track  mode,  but  remains  for  duration 
of  missile  launch 

Missile  Azimuth 

0-  to  360-deg 

Vector  to  threat  or  missile  azimuth  (A/A  and  S/A}- 
launch  mode  only 

Missile  "Tima  to 
Impact" 

,  0  to  x  tecs 

Boxed  readout  of  time  to  mistile  impact 
in  seconds  -  launch  mode  only 

Figure  8. 1-3.  HUD  Threat  Information  Coding 
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For  weapon  delivery,  an  ”X"  was  added  to  the  circle  of  the  airplane  symbol  when  the 
aircraft  entered  the  weapon  release  envelope.  When  the  pilot  pressed  the  weapon 
release  button  on  the  control  stick  to  hand  the  weapon  off  to  the  automatic  weapon 
fire  control  system,  the  "X”  flashed.  The  "X"  then  disappeared  when  the  weapon  was 
launched.  . 

Figure  8.1-2  shows  the  solid  "ground"  shading  of  the  color  presentation  mode.  In  the 
monochrome  version,  where  such  shading  was  not  possible,  radials  extended  from  a 
vanishing  point  on  the  horizon  in  the  "ground"  portion  of  the  display  to  distinguish  it 
from  the  "sky". 

8.1.2  Vertical  Situation  Display  (VSD) 

An  example  of  the  low  altitude  Vertical  Situation  Display  (VSD)  is  shown  in  Figure  8.1- 
4.  As  previously  discussed,  the  viewpoint  of  this  display  (6,000  feet  behind  and  1000 
feet  above  the  airplane)  differed  from  the  pilot's  eye  viewpoint  of  the  HUD.  The  VSD 
field  of  view  (90°  horizontal)  was  significantly  wider  than  the  HUD  field  of  view  (23°). 
Much  of  the  basic  VSD  symbology,  however,  was  similar  to  HUD  symbology.  The 
desired  flight  path  was  depicted  as  a  "ribbon  in  the  sky",  similar  to  the  "floor"  of  the 
HUD  "pathway".  The  ribbon  started  1,000  feet  in  front  of  the  airplane,  and  extended 
to  20,000  feet  ahead.  The  lines  across  the  ribbon  represented  a  distance  of  2,000  feet. 
Square  flags  marked  the  location  of  waypoints  and  a  triangular  flag  marked  the 
location  of  the  ground  target. 

Airspeed,  heading,  and  altitude  readouts,  in  boxes  at  the  top  and  sides  of  the  VSD, 
were  identical  to  those  on  the  HUD.  Terrain  information  on  the  VSD  was  similar  to 
terrain  information  on  the  HUD,  with  additional  three-dimensional  perspective 
information  available.  The  same  basic  terrain  features  were  depicted,  but  terrain 
above  current  airplane  altitude  was  colored  brown  to  distinguish  it  from  terrain 
currently  below  airplane  altitude,  which  was  colored  green.  In  the  monochrome 
version,  "above  altitude"  terrain  was  solid  white,  "below  altitude"  terrain  was  black. 
The  "ground"  area  of  the  display,  in  both  presentation  types,  included  a  grid  composed 
of  radials  extending  from  a  vanishing  point  on  the  horizon,  crossed  by  horizontal 
parallel  lines  which  got  progressively  closer  together  towards  the  horizon.  This  grid 
helped  to  give  a  perspective  to  the  VSD.  The  VSD  also  included  other  selected  terrain 


and  cultural  features  such  as  rivers  and  roads,  target,  city,  and  the  FEBA  (Forward 
Edge  of  the  Battle  Area). 

Information  about  threats  was  presented  pictorially  on  the  VSO  as  detailed  in  Figure 
8.1-5.  The  lethality  zones  of  SAM-type  threats  were  depicted  as  six-sided  cone-like 
objects  with  their  bases  up.  Their  position  was  shown  relative  to  terrain  features  and 
the  desired  flight  path.  There  were  two  generic  types  of  SAM's  represented:  high- 
altitude  and  low-altitude,  which  differed  primarily  in  the  height  and  radii  of  their 
lethality  zones.  Anti-aircraft  artillery  threats  (AAA's)  were  represented  by  truncated 
cone  figures,  base  downward,  that  were  much  shorter  than  the  SAM's. 

Threat  modes  (inactive,  search,  track,  launch)  were  coded  as  follows.  Inactive  threats 
(those  known  only  through  intelligence  data)  were  shown  in  a  transparent,  outline 
form.  When  sensor  data  indicated  an  active  threat  in  search  mode,  the  threat  was 
depicted  in  solid  color:  SAM's  had  an  inner  red  (high  lethality)  area  surrounded  by  a 
yellow  (lower  lethality)  area.  In  the  monochrome  version,  the  high  lethality  area  was 
solid  white,  while  the  surrounding  lower  lethality  area  was  black  with  white  outlines. 
AAA's  were  solid  red,  or  solid  white  in  the  monochrome  version.  The  location  (but  not 
the  lethality  areas)  of  air-to-air  threats  was  represented  by  a  red  triangle. 

For  threats  in  track  mode,  a  "tractor  beam"  was  added  that  connected  the  threat  to  a 
circle  around  the  own  aircraft  symbol.  In  launch  mode  (possible  only  after  the  aircraft 
had  entered,  a  threat  lethality  envelope)  the  threat  lethality  envelope  symbology 
disappeared  and  was  replaced  by  a  small  "jewel  light"  at  the  threat  site.  (For  air-tO- 
air  threats,  the  threat  location  triangle  remained  the  same  for  track  and  launch 
modes.)  A  flashing  tractor  beam  connected  the  circled  aircraft  symbol  to  the  "jewel 
light"  for  as  long  as  the  threat  remained  in  launch  mode.  Even  if  the  site  of  the  threat 
passed  out  of  the  VSO  field  of  view,  the  flashing  tractor  beam  remained,  and  the 
"jewel  light"  was  pegged  to  the  side  of  the  display  at  the  end  of  the  flashing  tractor 
beam.  For  both  track  and  search  modes,  the  circle  around  the  aircraft  symbol  was 
color  coded  to  reflect  the  status  of  countermeasures:  yellow  for  effective,  red  for 
ineffective  or  depleted.  In  the  monochrome  version,  white  outline  represented 
effective  countermeasures,  and  solid  white  represented  ineffective  or  depleted 
countermeasures* 
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Coding  in  Thraat  Warning  Study 

Threat  Typa 

Surface-to-air 

SAM  A 

'‘High  altitude"  SAM  -  tall  car# 

SAMB 

"Low  altitude"  SAM  -  shorter,  wider  cane 

AAA 

Inverted,  truncated  cone 

Air-to-air 

— ' 

Triangle  (red/white) 

Thraat  Lathality 

Surface- to -air 

Low 

Yellow/white  outline 

High 

Red/white 

Thraat  Mod* 

Air-to-air. 

Inactiva  (prefariefed) 

Outline  only 

surface-to-air 

Saarch 

Outline  plus  color/whita  fill 

Track 

Tractor  beam  connect*  aircraft  symbol 
(plus  circle)  to  thraat 

Launch 

Flashing  tractor  beam  connects  aircraft  symbol 
to  "jewel  light"  at  thraat  sits  (S/A) ;  dr  to  A/A 
thraat  position 

Countermeasures  Statu* 

Aircraft  circfe: 

Surfaao-to-ak 

Effactiva 

Yellow  fill/white  outline 

-ft-  ,a|.  , 

rVOt  ITTKllVI 

Rad  fill/white  «H 

The  NAV  mode  of  the  VSD  was  a  conventional  EADI  format.  It  included  an  airplane 
symbol,  horizon,  pitch  ladder,  and  roll  indicator,  as  well  as  the  airspeed,  heading  and 
altitude  boxes  found  on  the  TF/TA  (low  altitude)  version. 

8.1.3  Horizontal  Situation  Display  (HSD) 

The  low  altitude  Horizontal  Situation  Display  (HSD)  gave  a  plan  view  of  the  plane's 
position  and  desired  flight  path  relative  to  terrain  features  and  threats,  as  shown  in 
Figure  8.1-6.  Terrain  above  aircraft  altitude  was  colored  brown:  mountain  "peaks" 
were  four-  or  six-sided  brown  ("above  altitude")  figures  against  a  green  background  (in 
the  color  fill  version).  Threats  had  a  red  (high  lethality)  zone  surrounded  by  a  yellow 
(lower  lethality)  zone.  The  flight  route  was  represented  by  a  line  that  connected 
numbered  waypoint  boxes.  The  airplane  was  always  in  the  center  of  the  display,  track 
up.  The  airplane  symbol  remained  stationary  and  the  other  symbology  moved  relative 
to  it.  The  current  track  was  given  at  the  top.  The  pilot-selectable  range  was  given  at 
the  lower  left  (40,  80,  160  or  320  NM  from  the  top  to  the  bottom  of  the  display). 
Optional  time  and  distance  information  to  the  next  waypoint,  to  the  next  target,  or  to 
home  base  was  displayed  at  the  bottom  when  selected  by  the  pilot.  Other  terrain  and 
cultural  features  were  also  represented. 

Threat  information  was  presented  in  plan  view  on  the  HSD  with  coding  similar  to  that 
used  on  the  VSD-  Inactive  threats  were  shown  in  outline  form.  Threats  in  search  mode 
were  in  solid  color  (or  white)  and  a  tractor  beam  was  added  for  track-mode  threats. 
Unlike  coding  oh  the  VSD,  tractor  beam  di<L  not^flash  in  launch  mode.  Instead,  a 
moving  circle  marked  the  current  position  of  the  missile.  HSD  threat  coding  details 
are  summarized  in  Figure  8.1-7. 

The  high  altitude  NAV  mode  of  the  HSD  showed  the  flight  path,  waypoints,  and  other 
cultural  and  terrain  features,  but  the  above  altitude/below  altitude  terrain  distinction 
was  no  longer  relevant;  the  background  was  black. 

The  pilot  could  select  among  four  ranges  (40,  80,  160  or  320  NM)  for  the  NAV  mode  of 
the  HSD,  using  the  switches  to  the  left  of  the  HSD.  Time  and  distance  information 
could  also  be  selected,  along  with  optional  fuel  range  rings,  using  the  same  group  of 
switches.  The  fuel  range  rings  showed  the  flight  distance,  available  with  the  remaining 
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fuel,  for  both  present  configuration  (inner  ring)  and  maximum  range  configuration 
(outer  ring). 

8.1.4  Horizontal  Situation  Display  -  Air-to-Air  Mode 

Figure  8.1-8  shows  the  Air  to  Air  (A/A)  mode  of  the  HSD.  Each  sector  represented  the 
missile  "lock  on"  range  for  an  airplane  located  at  the  apex,  and  flying  in  the  direction 
of  the  long  axis  of  the  wedge.  "Owr."  aircraft  sector  was  white;  those  of  "enemy” 
aircraft  was  red  and  yellow.  The  inner  (red)  and  outer  (yellow)  portions  of  the  sectors 
represented  high  and  lower  probability  of  kill  areas.  As  in  the  other  HSD  modes,  the 
display  was  always  track  up  for  "own"  aircraft,  with  a  heading  box  at  the  top.  Range 
rings  represented  distances  of  40  and  80  miles,  and  the  FEBA  v/as  depicted  when  it  was 
within  the  display  range.  The  pilot-selectable  range  (40,  80,  16Q  or  320  NM)  was 
displayed  in  the  lower  left  corner. 

Air-to-air  threat  type,  number,  lethality  (p^)  zones  and  mode  information  were  also 
presented  as  detailed  in  Figure  8.1-9.  Air-to-air  threat  modes  were  analogous  to  air- 
to-ground  threat  modes:  outline,  fill,  connecting  tractor  beam,  and  tractor  beam  to 
jewel  light  at  threat  location  ('vith  missile  position  indicated  by  a  moving  circle) 
represented  inactive  (prebriefed),,  search,  track  and  launch  modes,  respectively.  The 
number,  type  and  altitude  difference  (from  "own"  aircraft,  in  thousands  of  feet)  were 
given  in  alphanumerics  in  a  box  at  the  apex  of  the  air-to-air  threat  sector. 
Abbreviated  air-to-air  threat  coding  was  also  given  in  the  other  modes  of  the  HSD 
(NAV  and  TF/TA).  In  these  formats,  an  air-to-air  threat  was  represented  by  an 
isoCeles  triangle  pointing  in  the  direction  of  the  "bogie's"  flight. 

8.2  System  Status  and  System  Advisory  Displays 

The  Multipurpose  Displays  (MPDs),  located  on  either  side  of  the  HSD,  were  used  to 
display  engine  and  system  status  information  and  to  program  and  display  stores 
options,  MPD  mode  selection  switches  were  located  in  a  group  above  each  MPD.  The 
left  MPD,  used  primarily  for  stores  programming  and  status,  was  oriented  with  its  long 
axis  horizontal.  Display  programming  switches  were  located  along  the  top  and  bottom 
of  this  MPD.  The  right  MPD,  used  for  engine  and  system  status  information,  had  its 
long  axis  oriented  vertically,  with,  display  programming  switches  cm  either  side  (not 
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used  during  this  simulation).  The  MPD  formats  used  in  the  threat  warning  study  were 
unaltered  from  those  in  the  basic  study,  described  in  Section  3.4. 

During  the  course  of  a  mission,  the  pilot  would  typically  have  the  engine  status  format 
displayed  on  the  right  MPD,  and  the  currently  selected  stores  option  on  the  left  MPD 
(initially  programmed  before  the  flight  began).  The  other  MPD  formats,  while 
available,  were  not  exercised  in  the  threat  warning  study. 


9.0  TEST  PLAN  -  THREAT  WARNING  STUDY 

The  test  plan  for  the  threat  warning  study  was  derived  from  the  prime  objectives  for 
Phase  D.  They  were  to  develop  a  candidate  concept  for  pictorial  threat  displays,  both 
air-to-air  and  surface-to-air;  evaluate  the  usability  and  acceptability  of  the  candidate 
threat  warning  formats;  and  determine  if  the  degree  of  usability  and  acceptability  of 
the  candidate  display  concept  is  a  function  cf  the  two  basic  display  presentation 
modes:  color  and  monochromatic. 

The  intent  was  to  provide  an  operationally  valid  evaluation  under  conditions  of 
moderate  experimental  control.  To  this  end,  current  military  fighter  or  attack  pilots 
were  recruited  as  evaluators  and  a  within -subjects  factorial  design  was  selected.  The 
test  procedure  provided  an  intensive  training  and  practice  period  before  testing  began. 
The  test  scenario  exposed  the  salient  features  of  the  formats.  Data  were  collected  to 
support  responses  to  the  program  objectives. 

9.1  Test  Subjects 

A  total  of  twelve  (12)  pilots  served  as  subjects  in  the  study.  Five  pilots  were  from 
McChord  AFB  and  seven  from  Whidbey  Island  NAS.  The  Air  Defense  Command  pilots 
from  McChord  were  current  in  the  F-106  airplane,  T-33  or  F-9  while  the  pilots  from 
Whidbey  were  current  in  either  the  A-6,  EA-6B  or  the  P-3.  The  special  qualifications 
of  the  pilots  are  listed  in  Table  9.1-1.  The  pilots  had  an  average  age  of  3C  years  and 
an  average  of  1,329  flying  hours  in  a  variety  of  jet  aircraft.  The  pilots  with  F-4  and 
F-106  experience  had  used  HUD'S.  Those  with  F-9,  or  A-6  experience  had  used  threat 
warning  devices.  The  A-6E  pilots  had  used  a  cathode  ray  tube  VDL  None  of  the  pilots 
who  served  as  subjects  in  the  threat  warning  study  had  participated  in  the  basic  study. 

Names,  telephone  numbers,  and  home  organizations  for  all  participating  pilots  were 
made  available  to  Boeing  approximately  two  weeks  prior  to  the  scheduled  test  date.  A 
training  package,  designed  to  familiarize  subjects  with  the  test  objectives,  the  test 
conditions,  and  the  simulation  facility,  was  forwarded  to  all  pilots  for  review  prior  to 
the  simulation. 


Table  9. 1-1:  Pilot  Qualifications 


Mean 

Range 

Age: 

30 

24-36  yr*. 

Total  Jet  Hour*: 

1,524 

465-2,600  hr*. 

OPERATIONAL  EXPERIENCE: 

Ever  Used  7 

Pilot  No. 

Brandi 

Type  rf  Aircraft 

HUD 

Threat 

Warning 

CRT 

1 

AF 

F-4,  T-38,  T-37 

F-4  F-4 

F-15  (Sim) 

No 

2 

N 

,  P-3,  S-2,  A-4,  F-4,  T-28 

No 

F-4 

No 

3 

N 

A-6E,  KA-6D,  TA-4J,  T-2C 

T-34C 

No 

A-6E 

A-6E 

4 

N 

A-6E,  A-4,  T-2 

No 

A-6E 

A-6E 

5 

AF 

T-33,  T-38,  T-37 

No 

No 

No 

6 

AF 

F-106,  T-33,  T-38,  T-37 

F-106 

T-33 

No 

7 

N 

A-6,  A-4,  T-2,  F-14 

A-7 

«Skn) 

A-6 

A-6 

8 

N 

A-6,  A-4,  T-2,  T-28,  T-39 

No 

A-6 

A-6 

9 

AF 

F-106.T-38,  T-33.  T-37 

F-106 

No 

No 

10 

AF 

F-108.T-33,  T-38.  T-37 

F-106 

No 

No 

11 

N 

EA-6B.TA-4J.  T-2.  T-28 

No 

EA-6B 

No 

12 

N 

EA-68,  A-4,  T-2,  T-34 

No 

EA-68 

No 

9.2  Test  Design 

The  experimental  design  selected  for  this  study  was  determined,  as  in  the  basic  study, 
by  the  desire  for  economical  use  of  the  operational  evaluator  pilots  and  by  a 
requirement  to  minimize  the  effect  of  individual  differences.  For  these  reasons,  a 
within-subjects  design  was  used  in  this  threat  warning  study,  as  in  the  basic  study. 
There  was  one  independent  variable:  display  presentation  mode.  It  had  two  levels: 
monochromatic  and  color.  Display  presentation  mode  was  selected  as  an  independent 
variable  in  order  to  determine  whether  the  use  of  more  expensive  color  displays  would 
be  justified  by  increased  usability  and  pilot  acceptance.  Each  subject  was  tested  in 
both  treatment  conditions  of  the  experiment.  Therefore,  error  variability  due  to 
individual  differences  between  groups  of  subjects  did  not  differentially  affect  the 
performance  measures.  This  design  also  minimized  the  number  of  subjects  required 
and  provided  more  data  per  subject  than  any  other  design.  Training  requirements  were 
reduced  since  skills  learned  under  initial  training  trials  transferred  to  all  subsequent 
conditions. 
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9.3 


Test  Procedures 


Before  reporting,  pilots  had  access  to  and  were  asked  to  study  a  "flight  manual" 
(Reference  5)  which  described  the  simulator  cockpit,  the  formats  to  be  evaluated  and 
the  missions  to  be  flown.  The  test  subjects  reported  to  the  Kent  Space  Center  in 
pairs.  Two  days  were  required  to  process  each  pair  of  subjects  through  the  entire 
training  and  test  sequence.  The  daily  test  schedule  shown  in  Table  9.3-1  presents  the 
sequence  of  events  for  the  first  two  pilots.  The  same  basic  sequence  of  events  was 
used  for  ail  subsequent  test  days  except  that  the  order  of  the  treatment  conditions  was 
varied. 

Each  test  day  began  by  bringing  the  simulator  on-line  and  checking  to  see  that  all 
equipment  was  operating  properly.  Any  detected  deficiencies  were  corrected  before 
the  first  practice  or  test  trial  began. 

On  the  first  day,  pilots  were  given  an  introduction  to  the  multimission  simulator  while 
the  cab  was  being  initialized  for  the  first  practice  trial.  Following  the  introduction, 
the  pilots  were  briefed  on  the  genera!  cockpit  layout,  the  formats  to  be  evaluated,  the 
operational  procedures  to  be  used,  and  the  flight  plan  for  the  simulated  mission. 
Static  pictorial  representations  on  film  transparencies  were  used  to  introduce  display 
formats  that  would  be  encountered  during  the  practice  and  test  trials.  The  basic 
features  of  each  display  and  the  mission  profile  were  described  to  the  subjects.  Then 
the  pilots  were  taken  to  the  simulator  cab  for  familiarization  and  training. 

Each  pilot  flew  four  practice  trials  using  a  training  mission  with  the  same,  basic 
mission  profile  and  flight  duration  used  in  the  test  missions.  The  training  mission  was 
flown  twice  with  color  and  twice  with  monochrome  displays.  At  the  beginning  of  each 
practice  and  test  trial,  the  pilot  completed  the  same  preflight  checklist  used  in  the 
basic  study.  After  the  practice  trials,  each  pilot  flew  six  twenty-four  minute  test 
trials,  three  with  each  display  presentation  mode.  The  presentation  order  for  levels  of 
the  display  mode  and  mission  is  shown  in  Table  9.3-2.  Entries  in  the  body  of  the  table 
are  the  ordinal  numbers  of  each  subject's  trials.  As  can  be  seen,  the  orders  of  display 
mode  and  mission  number  were  counterbalanced  over  the  twelve  subjects. 
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The  pilots  wore  a  headset  with  microphone  during  ail  trials.  They  were  asked  for 
verbal  reports  of  changes  in  threat  status.  These  responses  were  recorded  on  the  audio 
channel  of  the  video  tape  for  subsequent  analysis.  Required  switch  hit  responses  were 
recorded  automatically  by  computer  on  magnetic  tape  and  printed  out  at  the  end  of 
each  days  run.  Both  the  videotape  and  the  computer  recorded  data  had  deck  data 
(hours,  minutes,  seconds,  video  frame,  and  date)  recorded. 


Table  9.3- 1.  Typical  Daily  Test  Schedule 


0730-1000 

DAY  1 

Bring  simulator  on  tine  and  checkout 

0800-0930 

Introduction  to  facility  and  pre-flight  briefing 

0930-1020 

Cockpit  familiarization 

1020-1055 

Practice  trial,  color,  subject  1 

1055-1130 

Practice  trial,  color,  subject  2 

1130-1230 

Lunch  Break 

1230-1305 

Practice  trial,  color,  subject  1 

1305-1340 

Practice  trial,  color,  subject  2 

1340-1415 

Practice  trial,  monochrome,  subject  1 

1415-1450 

Practice  trial,  monochrome,  subject  2 

1450-1525 

Practice  trial,  monochrome,  subject  1 

1525-1300 

Practice  trial,  monochrome,  subject  2 

1600-1635 

Test  trial,  color,  subject  1 

1635-1710 

Test  trial,  color,  subject  2 

0730-0830 

DAY  2 

Bring  simulator  on  line  and  checkout 

0830-0906 

Test  trial,  monochrome,  subject  1 

0905-0940 

Test  trial,  monochrome,  subject  2 

0940-1015 

Test  trial,  color,  subject  1 

1015-1060 

Test  trial,  color,  subject  2 

1060-1126 

lest  trial,  m  >nochrome,  subject  1 

1125-1225 

Lunch  Break 

1225-1300 

Test  trial,  monochrome,  subject  2 

1300-1326 

Test  trial,  color,  subject  1 

1325-1410 

Test  trial,  color,  subject  2 

1410-1445 

Test  trial,  monochrome,  subject  1 

1445-1520 

Test  trial,  monochrome,  subject  2 

1520-1720 

Debriefing,  questionnaire  and  taped  interview 

Table  9.3-2.  Sequence  Of  Conditions  For  Test  Trials 


Color 

Monochrome 

Subject 

Mission  1 

Mission  2 

Mission  3 

Mission  1 

Mission  2 

Mission  3 

1 

1 

3 

6 

1 

4 

1 

5 

2 

6 

1 

3 

6 

1 

7 

1 

8 

1. 

9 

1 

6 

10 

2 

•EH 

1 

11 

6 

1 

12 

6 

4 

■■ 

3 

1 

5 

9.4  Mission  Scenario 

The  flight  plan  for  the  simulated  mission  is  shown  in  Figures  9.4-1  and  9.4-2.  Table 
9.4-1  defines,  pertinent  aircraft  conditions  and  mission  events  for  each  leg  of  the  flight 
plan.  The  mission  covered  approximately  240  nautical  miles.  During  each  trial  the 
pilot  experienced  manual  and  coupled  flight,  air-to-air  and  air-to-ground  weapon 
deliveries,  and  airspeeds  from  500  knots  to  Mach  1.6  under  Instrument  Meteorological 
Conditions  (IMC). 


Table  9.4-1.  Flight  Plan 


Leg 

Weather 

Control 

Altitude  IAGL) 

Airspeed 

Distance 

time 

A  -  Approach  FEBA 

IMC 

Autopilot 

20K  ft 

SOOKts 

28  nm 

3.35  min 

B  •  Penetration 

IMC 

Autopilot 

20K  ft  •  200  ft 

500  Kts 

31.5  nm 

3.79  min 

C  •  Approach  IP 

IMC 

Manual 

200  ft 

500/565  Kts 

42  nm ' 

4.59  min  < 

O •  Target 

IMC 

Manual 

200  ft 

5b0  Kts 

2.1  nm 

2.29  min 

E  -  Withdrawal 

IMC 

Manual 

200  ft 

600  Kts 

33.5  nm 

3.35  min  ' 

F  •  Escape 

IMC 

Autopilot 

200  ft  ■  20  Kft 

Mach  1.6 

'  46.9  nm 

2.61  min 

G  ■  Return 

IMC 

Autopilot 

20K  ft 

600  Kcs 

38-5  nm 

3. 85  .min 

241.40  nm 


23.93  min 


>  K~' 


'?■  ■';■  * 1 
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Just  prior  to  initiating  the  mission,  the  piiot  exercised  the  preflight  checklist.  He  used 
the  left  Multi-Purpose  Display  (MPD)  to  program  the  weapons  options.  The  initial 
weapons  format  showed  all  stores  in  the  inventory  (2  ECM  pods,  2  napalm  canisters,  2 
air-to-air  missiles,  12  MK  82  bombs),  but  nothing  was  selected  or  fused. 

The  missions  began  with  the  aircraft  on  autopilot  at  20,000  feet  and  on  a  heading  of 
330  degrees.  Cruise  speed  was  500  knots  to  the  Forward  Edge  of  the  Battle  Area 
(FEBA).  The  aircraft  had  just  completed  aerial  refueling  and  was  proceeding  inbound 
to  the  pre-assigned  target  area,  a  railroad  switching  yard. 

The  pilot  encountered  enemy  aircraft  proceeding  outbound  from  the  FEBA  on  an 
intercept  course.  The  pilot  set  the  ECM  pods  to  ON,  selected  air-to-air  missiles  as  the 
active  weapon  option  and  turned  the  master  arm  on. 

The  air-to-air  format  was  automatically  displayed  on  the  HSD  when  the  enemy 
aircraft  was  detected.  The  enemy  aircraft  appeared  at  the  top.  of  the  HSD  Air/ Air 
mode  format.  An  "X"  appeared  in  the  middle  of  the  HUD  airplane  symbol  when  an 
enemy  aircraft  was  within  radar  range.  At  this  point,  the  pilot  could  press  the  weapon 
release  trigger  to  hand  the  missile  off  to  the  automatic  weapon  fire  control  system, 
and  the  "X"  then  flashed.  A  missile  was  launched  and  the  enemy  aircraft  destroyed* 
When  the  missile  was  fired,  the  flashing  nXn  disappeared.  At  this  time  the  piiot  set 
the  ECM  pods  to  standby  and  selected  MK  82  bombs  as  the  active  weapons  option. 

L£*B 

The  aircraft  descended  from  20,000  feet  to  200  feet  above  terrain  immediately  after 
the  air-to-air  engagement. 

During  this  leg.  the  pilot  was  sometimes  alerted  to  a  heat  seeking  missile  fired  from 
behind.  The  pilot  was  required  to  activate  an  "evade”  switch  during  an  appropriate 
time  window,  3-5  seconds  from  missile  impact,  to  simulate  initiation  of  an  appropriate 
evasive  maneuver. 
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LegC 


The  autopilot  established  a  heading  of  040  degrees  at  the  beginning  of  Leg  C.  After 
this  heading  was  captured,  the  pile.;  disengaged  the  autopilot  and  flew  manually, 
adjusting  the  power  to  maintain  approximately  550  knots.  The  VSD  and  HSD  formats 
were  used  to  observe  the  position  of  enemy  ground  threats  during  the  approach  to  the 
target  area. 

The  pilot's  primary  task  during  legs  C  through  F  was  to  use  the  VSD  and  HSD  formats 
to  minimize  exposure  to  enemy  ground  threats.  Displayed  ground  threats  included 
both  inactive  and  active  threats.  On  the  VSD,  inactive,  search,  and  track  mode 
threats  remained  on  the  displays  as  long  as  they  were  within  the  field  of  view.  On  the 
HSD,  active  and  inactive  threats  were  displayed  whenever  they  were  within  the 
selected  range.  When  a  threat  was  in  track  mode,  and  countermeasures  were 
ineffective  or  depleted  (as  indicated  by  the  color  of  the  circle  around  the  airplane 
symbol),  the  pilot  activated  a  navigation  update  switch  that  fed  the  new  threat  data  to 
the  navigation  computer  so  that  the  flight  track  and  the  displayed  flight  channel  could 
be  offset  to  take  the  best  evasive  action.  When  a  threat  was  in  laurch  mode,  and  the 
countermeasures  were  ineffective  or  depleted,  the  pilot  activated  an  evade  switch 
(located  on  the  control  stick)  to  simulate  initiation  of  a  "jink"  evasive  maneuver. 

hsLQ. 

The  pilot  manually  turned  onto  a  heading  of  315  degrees  at  the  beginning  of  Leg  D. 
The  pilot  could  optionally  highlight  the  target  area,  a  railroad  switching  yard,  as  the 
aircraft  approached  the  weapon  release  point.  All  MK  82  bombs  were  delivered  on  a 
single  pass  over  the  target.  Entry  into  the  weapon  delivery  envelope  was  cued  by  the 
appearance  of  an  "X”  on  the  Head-Up  Display  (HUD). 

Le&E 

The  aircraft  was  flown  manually  to  a  heading  of  250  degrees  at  waypoint  5  and  speed 
was  increased  to  600  knots. 
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Le&l 


The  autopilot  was  engaged  at  waypoint  6  after  crossing  the  FEBA  outbound.  The 
aircraft  captured  a  heading  of  190  degrees  and  the  afterburner  was  used  during  a  Mach 
1.6  escape  to  cruising  altitude  of  20,000  feet.  Speed  was  reduced  to  600  knots  at  top 
of  climb.  On  some  flights,  the  pilot  was  alerted  to  a  heat  seeking  missile  from  behind, 
and  responded  by  activating  the  "evade"  switch.  In  some  missions,  an  enemy  aircraft 
approached  for  an  engagement  in  this  leg.  The  pilot  disengaged  the  autopilot  and 
engaged  the  enemy  in  manual  flight. 

lsblg 

The  aircraft  autopilot  captured  a  heading  of  126  degrees  at  the  beginning  of  Leg  G.  A 
final  enemy  air-to-air  engagement  sometimes  occurred  in  this  leg,  if  one  had  not 
occurred  in  Leg  F.  The  mission  ended  as  soon  as  the  final  air-to-air  engagement  was 
completed. 

9-5  Threat  and  Mission  Variations 

Four  missions  were  developed  for  this  study,  one  for  pilot  practice  and  three  for 
testing.  These  missions  varied  only  in  the  location  and  behavior  of  the  air-to-air  and 
surface-to-air  threats.  Figures  9.5-1  through  9.5-4  show  the  threat  beddown  locations 
for  the  four  missions.  In  those  figures,  the  circles  around  the  surface  threat  locations . 
represent  maximum  launch  range  at  ground  level.  For  air-to-air  threats,  the  bogies' 
location  and  the  location  of  our  aircraft  at  the  start  of  the  engagement  are  shown. 

The  threat  encounters  for  each  mission  were  drawn  from  a  catalog  of  twenty-four 
surface  threats  and  twelve  air  threats.  /Among  the  surface  threats,  seven  were  anti¬ 
aircraft  artillery  sites,  eight  were  low  altitude  SAM's  and  nine  were  high  altitude 
SAM's.  The  air  threats  included  four  which  closed  and  could  be  fired  upon  without 
maneuvering,  four  which  closed  but  required  maneuvering  to  hit,  and  four  incoming 
air-launched  missiles  which  had  to  be  evaded. 

Table  9.5-1  shows  the  threat  composition  of  each  of  the  four  missions.  Each  mission 
used  sixteen  of  the  twenty-four  threats,  chosen  so  that  each  threat  was  used  a  total  of 
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Figure  9.5 -  1.  Threat  Locations  for  the  Practice  Mission 
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figure  9.5-2.  Threat  Locations  lor  Tost  Mission  No.  1 
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Test  Data  Collection 


As  in  the  basic  study,  both  objective  performance  and  subjective  opinion  data  were 
collected.  These  data  addressed  the  questions  of  overall  suitability  of  the  pictorial 
.  format  display  concepts,  the  selection  between  color  and  monochrome 
implementations  of  those  concepts,  and  identification  of  particular  format  weaknesses 
which  could  be  corrected  in  revised  formats. 

Ten  measures  of  pilot  performance  were  collected.  These  measures  provided 
quantitative  data  on  the  pilot's  ability  to  use  the  pictorial  formats  in  the  two 
presentation  modes  to  accomplish  flight  path  control,  threat  detection  and  avoidance, 
and  weapon  delivery. 

9.6.1  Flight  Path  Control  Performance  Mean  ret 

During  three  manual  legs  of  the  mission,  one  of  the  pilot's  tasks  was  to  use  the  flight 
displays,  primarily  the  HUD,  to  maintain  the  desired  flight  path.  Airplane  position  was 
sampled  at  one-second  intervals  from  the  time  that  the  pilot,  on  cue,  selected  manual 
flight  until  the  time  he,  on  cue,  resumed  automatic  flight.  Four  measures  were 
derived  from  these  samples: 


RMS  vertical  deviation  -  RMS  (root  mean  squared)  error  in  the  vertical  or 
altitude  dimension 

RMS  lateral  deviation  -  RMS  (root  mean  squared)  error  in  the  lateral  or 
cross-track  dimension 

Percent  of  time  in  flight  path  channel  (130  feet  high  by  300  feet  wide) 
Percent  of  time  in  tight  channel  (73  feet  high  by  130  feet  wide) 
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Verbal  Reports 


Throughout  the  missions,  whenever  a  threat  went  into  an  active  mode  (searchj  track  or  j 

launch)  or  upgraded  from  one  active  mode  to  another  more  threatening  mode,  a  threat  j 

alert  symbol  appeared  on  the  HUD.  The  pilots  were  instructed  to  report  these  events  j 

immediately  with  several  items  of  information  available  from  one  or  more  of  the  three 
major  displays  as  discussed  in  Section  S.l.  Required  verbal  responses  are  given  in 
Table  9.6-1.  Three  measures  were  taken  of  these  verbal  responses. 

-  '  ’  ■  i 

Latency  of  verbal  response  -  Elapsed  time  from  appearance  of  threat  alert 

symbol  to  beginning  of  verbal  response  { 

I 

j 

•  ! 

Duration  of  verbal  response  -  Elapsed  time  from  beginning  to  end  of  verbal 
response 

-  Correctness  of  verbal  response  -  Was  response  correct  and  complete  or 
not? 


TaMr  9.6- 1.  Rtquind  Verba/  Rmponm 
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Motor  Responses  (Switch  Hits) 


When  a  threat  went  into  track  or  launch  mode  and  countermeasures  were  not 
effective,  tie  pilots  were  instructed  to  respond  with  an  appropriate  evasive  action. 
When  a  surface  threat  went  into  track  mode,  and  countermeasures  were  not  effective, 
he  was  to  press  the  "Nav  Update"  switch.  If  done  in  time,  this  would  cause  the 
computer  to  calculate,  the  displays  to  show,  and  the  airplane  to  fly,  a  safe  route 
around  the  threat  and  back  to  the  original  flight  plan.  As  soon  as  any  threat  launched, 
and  countermeasures  were  not  effective,  the  pilot  was  to  press  the  "Evade"  switch. 
For  AAA  threats,  this  was  to  be  done  immediately;  and  for  missiles  (SAM  or  AAM)  this 
was  to  be  done  from  three-to-five  seconds  before  projected  missile  impact.  The 
measures  taken  on  these  responses  were* 

_  Latency  of  Nav  Update"  response  -  Elapsed  time  from  threat  going  into 
track  mode  to  switch  hit 

-  Switch  hit  errors  -  Inappropriate  or  missed  "Nav  Update"  or  "Evade"  switch 
hit 

9.A.*  Weapon  Release  Measures 

The  objective  of  these  missions  was  a  bomb  drop  on  a  target.  The  pilots  were 
instructed  to  press  the  weapon  delivery  switch  as  soon  as  the  HUD  weapon  release 
symbol  indicated  entry  into  the  bomb  drop  window.  They  were  also  instructed  to  press 
the  weapon  delivery  switch  in  air-to-air  engagements  (two  opportunities  per  mission) 
as  soon  as  the  adversary  entered  the  AAM  delivery,  envelope  and  the  HUD  weapon 
release  symbol  was  displayed.  Response  latencies  to  these  w  eapon  release  situations 
were  measured. 

9^3  Pilot  Opinion  Measures 

A  questionnaire  (Appendix  C)  was  administered  after  the  simulation  trials  were 
completed.  It  was  divided  into  three  parts.  In  the  first  part,  ten  attributes  or 
characteristics  of  display  symbology  were  defined.  Then  the  pilots  were  asked  to  rate 
individual  elements  of  the  HUD,  VSD  and  HSD  symbology  on  a  seven  point  scale,  from 


"very  good"  to  "very  poor",  against  those  attributes.  The  attributes  are  listed  and 
defined  below. 

Usability  in  Color.  How  usable  was  this  display  element  in  the  color  display  mode? 

Conspicuousness  in  Color.  How  easy  was  it  to  see  this  display  element  in  the  color 
mode? 

Usability  in  Monochrome.  How  usable  was  this  display  element  in  the  monochrome 
display  mode? 

Conspicuousness  in  Monochrome.  How  easy  was  it  to  see  this  display  element  in  the 
monochrome  display  mode? 

Location.  Is  this  format  element  in  the  right  place  and  on  the  right  display  (HUD, 
VSD,  HSD)? 

Meaning.  How  clear  or  obvious  is  the  meaning  of  this  format  element? 

Precision.  Does  this  format  element  convey  its  information  with  the  appropriate  level 
of  precision? 

Timeliness.  Is  this  . format  element  available  to  you  at  the  right  time  and  for  the  right 
duration? 

Training.  How  easily  could  this  format  element  be  learned? 

Workload.  Does  this  format  element  contribute  to  workload  or  relieve  it?  ’ 

In  the  second  part  df  the  questionnaire,  pilots  were  asked  to  rate  the  usability  of 
monochromatic  and  color  presentation  modes  for  particular  aspects  of  the  HUD,  VSD 
and  HSD,  and  for  crosschecking  certain  information.  Again  a  seven  point  scale  was 
used,  this  time  from  "very  easy"  to  "very  difficult". 

The  third  part  of  the  questionnaire  was  a  list  of  ten  open-ended  questions.  The  pilots 
spoke  their  responses  into  a  tape  recorder  and  the  resulting  records  were  transcribed 
for  analysis. 
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10.0 


RESULTS  -  THREAT  WARNING  STUDY 


10.1  Pilot  Performance  Data 

Table  10.1-1  summarizes  the  ten  objective  measures  of  pilot  performance  that  were 
collected  in  the  threat  warning  study.  The  data  shown  are  ai!  mean  values  taken 
across  the  twelve  subjects  and  the  three  missions,  repeated  once  for  color  and  again 
for  the  monochrome  display  version.  Although  only  two  of  these  measures  (RMS 
vertical  error  and  percent  correct  verbal  reports)  were  statistically  significant,  all  ten 
;  showed  an  advantage  for  the  color  display  version.  The  individual  response  measures 
are  discussed  in  the  paragraphs  which  follow. 

10.1.1  Flight  Path  Control 


Table  10.1-2  gives  means  for  each  pilot  across  the  three  missions  on  each  of  the  lour 
measures  of  flight  path  control.  At  the  bottom  of  the  table  the  means  and  standard 
deviations  across  pilots  are  given  as  are  the  four  paired  measures  t-tests.  With  11 
degrees-of-freedom,  the  .05  level  criterion  for  significance  of  a  one-tailed  test  is 
2.202.  With  this  criterion,  only  the  difference  between  color  and  monochrome 
presentation  modes  for  RMS  vertical  error  was  significant.  All  four  differences, 
however,  are  in  the  expected  direction  with  color  better  than  monochrome. 


Table  10.  /•  1.  Objective  Performance  Summary 


' 

Color 

Monochrome 

RMS  vertical  error  (feat) 

257.1 

343.6 

RMS  lateral  error  (feet) 

1222.4 

1696.8 

Percent  time  in  150  foot  by  300  foot  window 

67.7 

64.2 

Percent  time  in  75  foot  by  ISO  foot  window 

44.7 

42.1 

Verbal  report  latency  (seconds) 

2.28 

2.35 

Verbal  report  duration  (teconds) 

3.83 

3.94 

Percent  correct  verbal  report* 

88.47 

86.69 

Navigation  update  latency  (seconds) 

7.17 

7.66 

Percent  correct  switch  hits 

88.83 

85.83 

Weapon  release  latency  (seconds) 

2.52 

2.64 

Table  10. 1-2.  Flight  Path  Control  Summary  Data 


Pilot 

RMS  (ft) 

Vertical  error 

HMS  (ft) 

Lateral  error 

Percent  time  in 

1 50  x  300  ft  window 

Percent  time  in 

75  x  150  ft  window 

Color 

Monochrome 

Color 

Monochrome 

Color 

Monochrome 

Color 

Monochrome 

1 

148.2 

816.3 

830.8 

81.4 

80.1 

mem 

56.2 

287.8 

1 

987.8 

913.0 

63.1 

64.1 

■SB 

37.5 

323.0 

1 

2607.2 

3025.1 

67.5 

67.3 

49.9 

45.4 

478.3 

398.0 

1892.0 

2382.1 

47.5 

54.6 

24.8 

29.7 

5 

310.6 

389.9 

772.6 

1188.2 

47.7 

52.4 

20.4 

24.4 

mm 

375.4 

742.6 

811.5 

983.1 

66.5 

35.5 

40.2 

33.9 

o 

289.2 

140.9 

1937.2 

671.0 

70.7 

73.9 

40.8 

48.9 

m 

381.5 

657.2 

3423.6 

77.7 

62.9 

54.7 

40.0 

mm 

407.1 

696.1 

1047.1 

80.3 

78.1 

65.1 

60.3 

10 

■Kiilill 

271.5 

677.6 

1579.0 

78.7 

75.3 

55.5 

45.4 

11 

303.1 

1704.5 

1629.2 

.  58.2 

59.4 

37.6 

35.4 

12 

185.0 

m 

1108.7 

2689.2 

73.3 

66.0 

52.9 

48.2 

Moan 

i 

1222.4 

1696.8 

67.7 

64.2 

42.1 

s.o. 

■Bai 

647.1 

945.1 

11.8 

12.6  ’ 

BESS 

10.0 

t 

j  -1.943* 

•1.673 

1.133 

1.408 

*  Significant,  p  <  .06 


10.1.2  Verbal  Reports 

Table  10.1-3  gives  the  mean  latency  and  duration  of  verbal  reports  for  each  pilot.  To 
make  the  values  as  comparable  as  possible,  the  subject  means  include  only  those  cases 
for  each  pilot  where  the  particular  reports  were  complete  and  correct  for  both  color 
and  monochrome  display  versions.  While  the  mean  difference  across  subjects  was  not 
significantly  different  for  either  latency  or  duration,  both  measures  showed  the  color 
display  version  to  be  slightly  better. 

Table  10.1-4  gives  percent  correct  verbal  reports  and  the  breakdown!  of  incorrect 
reports  into  corrected  errors,  uncorrected  errors,  and  omitted  or  incomplete  reports. 
By  Chi-square  test,  the  difference  in  frequencies  of  correct  reports  between  color  and 
monochrome  display  versions  was  significant  at  the  .05  level.  Again,  a  slight 
performance  improvement  was  found  with  the  color  formats.  The  most  common  errors 
in  these  verbal  reports  were  incomplete  or  omitted  reports.  Among  these,  the 
difference  in  frequencies  of  incomplete  or  omitted  reports  between  color  and 
monochrome  display  versions  was  significant  by  Chi-square  at  the  .05  level. 


Table  10. 1-3.  Verba /  Report  Latency  And  Duration  ( Seconds ) 


Verbal  Report 
Latency 

Verbal  Report 
Duration 

Pilot 

Color 

Mono 

Color 

Mono 

1 

2.23 

2.05 

3.51 

3.22 

2 

1.91 

1.64 

6.69 

6,59 

3 

2.21 

2.42 

3.12 

3.16 

4 

2.39 

2.74 

3.38 

3.27 

5 

2.01 

2.49 

5.27 

5.53 

6 

2.98 

3.1,8 

2.68 

2.88 

7 

2.08 

1.90 

4.29 

4.39 

8 

2.75 

2.84 

3.88 

4.02 

9 

2.03 

2.23 

3.82 

3.27 

10 

2.21 

1.87 

4.12 

4.69 

11 

2.24 

2.69 

2.17 

2.74 

12 

2.37 

2.16 

2.99 

3.46 

Mean 

2.28 

2.35 

3.83 

3.94 

S.D. 

0.31 

0.46 

1.21 

1.17 

t 

-.794 

-1.101 

Table  10. 1-4.  Percent  Correct  and  Incorrect  Verba!  Reports 


, 

Correct 

Corrected 

errors 

Unce*rected 

error* 

Omitted 

responses 

Color 

88.47 

2.30 

3.93 

5.30 

Monochrome 

86.69 

2.62 

3.97 

6.73 

X2,df-  1 

4.13‘ 

4.71‘ 

‘Significant,  p  <  .05 


10.1.3  Motor  Responses  and  Weapon  Release 

Table  10.1-5  gives  the  mean  latencies  for  navigation  update  and  weapon  release  switch 
hits.  As  with  the  verbal  report  latencies,  the  means  for  each  subject  include  latencies 
only  for  events  for  which  the  responses  were  correct  in  both  the  color  and 
monochrome  display  versions.  This  was  an  attempt  to  avoid  contamination  of  the 
latency  data  by  incorrect  responses.  For  both  these  measures,  the  mean  latency  was 
slightly  shorter  for  color  formats,  but  neither  was  statistically  significant. 


Table  10.1-6  gives  percent  correct  "navigation  update"  and  "evade"  switch  nix 
responses.  Incorrect  responses  are  broken  down  into  corrected  errors,  uncorrected 
errors  and  omitted  responses.  By  Chi-square  test,  the  difference  in  correct  responses 
between  colored  and  monochrome  display  versions  was  not  significant  at  the  .05  level. 


Table  10. 1-5.  Navigation  Update  and  Weapon  Release  Latencies  (Seconds) 


Pilot 

Navigation  update  latency 

Weapon  release  latency 

* 

Color 

Monochrome 

Color 

Monochrome 

1 

7.33 

6.65 

1.74 

2.84 

8.92 

6.23 

5.88 

2.11 

6.59 

6.32 

1.32 

2.48 

9.13 

7.85 

1.96 

2.17 

6.33 

10.91 

4.27 

2.80 

6.58 

8.54 

2.37 

2.90 

4.87 

6.85 

4.61 

2.09 

8 

8.39 

9.09 

2.23 

5.73 

9 

8.63 

7.26 

1.47 

1.41 

10 

5.85 

6.89 

1.24 

1.72 

11 

5.22 

5.71 

2.07 

3.13 

12 

8.19 

9.57 

1.07 

2.31 

Mean 

7.17 

7.66 

2.52 

2.64 

EH 

1.47 

1.57 

1.54 

1.09 

■5 

-.875 

-.218 

Table  10.1-6.  Percent  Correct  and  Incorrect  Switch  Hits 


Correct 

Corrected 

errors 

Uncorrected 

error* 

Omitted  or 
incomplete 

Color 

88.83 

■1 

1.33 

8.78 

Monochrome 

85.83 

H 

2.50 

10.0 

10.2 


Opinion  Data 


Appendix  A  contains  a  copy  of  the  questionnaire  with  mean  responses  for  the  rating 
questions.  In  general,  the  responses  were  fairly  tightly  grouped  and  quite  favorable. 

In  Figure  10.2-1,  ratings  of  HUD  symbology  elements  for  usability  in  color  and 
usability  in  monochrome  are  plotted  as  profiles.  With  the  exception  of 
waypoint/target  flags,  where  the  ratings  are  tied,  color  was  judged  slightly  better  than 
monochrome  for  all  the  symbology  elements  covered.  The  profiles  also  show  tliat, 
except  for  the  transitional  flight  director,  the  elements  were  judged  from  slightly 
good  to  very  good.  This  was  the  first  time  the  transitional  flight  director  was  included 
in  the  display,  and  its  implementation  was  incomplete  at  the  time  of  the  threat 
warning  study. 

Table  10.2-1  shows  a  correlation  matrix,  across  subjects  and  symbology  elements,  of 
attribute  ratings  for  the  HUD.  For  both  color  and  monochrome,  rated  usability  was 
most  highly  correlated  with  precision,  timeliness,  training  and  workload.  Caution 
should  be  exercised  in  interpreting  this  correlation  matrix  and  the  others  to  be 
presented.  The  characteristics  of  the  response  scale  and  the  tight  grouping  of 
responses  may  violate  some  of  the  assumptions  for  the  Pearson  product-moment 
correlation  coefficient.  However,  at  this  level,  that  statistic  provides  a  valid 
indication  of  relatedness. 

I 

Figure  10.2-2  profiles  the  color  and  monochrome  usability  ratings  for  the  VSD.  For 
each  of  the  symbology  elements,  color  was  rated  better  than  monochrome.  Even  in 
the  monochromatic  mode,  only  altitude  terrain  coding  wa!  rated  as  low  as  neutral. 

The  correlation  matrix  for  the  VSD,  Table  10.2-2,  shows  that  for  the  colored  symbols, 
usability  was  most  closely  correlated  with  conspicuous  less,  training,  workload  and 
precision.  For  the  monochromatic  symbols,  training,  ccmpicuousness,  precision  and 
timeliness  were  most  closely  correlated  with  usability. 

As  Figure  10.2-3  shows,  the  proxies  of  rated  usability  for  color  and  monochrome  in  the 
HSD  are  parallel  and  quite  close.  The  two  items  for  which  mean  ratings  were  tied, 
bogie  number  and  type  and  bogie  altitude,  are  both  numeric  rather  than  symbolic 
elements.  All  elements  in  both  display  modes  had  mean  ratings  better  than  neutral. 
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Search  mode  threat 
alert  symbol 


Track  mode  threat 
alert  symbol 


Time-to-missile 
impact  readout 


Threat  azimuth 
vector 


Pathway-in-the-sky 


Transitional  flight 
director 


Terrain  information 


Weapon  release  cue 


Waypcint/target 

flags 


Figure  10.2-7.  Usability  Ratings  for  HUD  in  Color  and  Monochrome 

The  correlation  matrix  shown  in  Table  10.2-3  shows  that  for  both  color  and 
monochrome,  usability  was  most  closely  correlated  with  conspicuousness,  training, 
meaning  and  precision.  The  HSD  correlations  are  generally  higher  than  comparable 
correlations  for  the  other  displays. 


The  results  of  the  second  part  of  the  questionnaire  are  plotted  in  Figure  10.2-4.  The 
pilots  were  asked  specifically  to  rate  the  two  display  modes  for  difficulty  in  extracting 
or  crosschecking  specific  information.  The  familiar  pattern  appears  *  all  items  were 
rated  neutral  or  better,  with  color  slightly  better  than  monochrome.  The  tie  was  for 
threat  type  in  the  air-to-air  HSD  mode,  which  was  presented  digitally. 
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Inactive  threat 
lethality  envelope 


Active  threat 
lethality  envelopes 


Track  mode  "tractor 
beam" 


Launch  mode 
symbology 


Countermeasures 
effectiveness  coding 


Ribbon-in-the-sky 


Altitude  terrain 
coding 


Geographic  and 
cultural  features 


Waypoint/target  flags 


In  color  and  monochrome,  these  display  element',  are: 

Good  Poor 

Very  Moderately  Slightly  Neutral  Slightly  Moderately  Very 


Figure  10.2-2.  Usability  Ratings  for  VSD  in  Color  and  Monochroma 
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In  color  and  monochrome,  these  display  elements  are: 
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Figure  10.2-3.  Usability  Ratings  for  HSD  in  Color  and  Monochrome 


In  color  and  monochrome,  it  was  this  easy  to  obtain  information: 
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HUD  threat  alert 


HUD  pathway 
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VSD  threat  type 


VSD  countermeasures 


VSD  ribbon 


VSD  terrain 


HSO  threat  mode 


HSD  threat  type 


HSD  countermeasures 
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Figure  1 0.2-4.  Eaaa  of  Uta  Judgments  for  Color,  and  Monochroma 
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7*6/»  10.2- 1.  Hoad  Up  Display  —  Corraiation  Matrix  of  Ratings 


Symbology  attributes 


A.  Usability  in  color 

B.  Conspicuousness  in  color 

C.  Usability  in  monochrome 

D.  Conspicuousness  in  monochrome 

E.  Location 

F.  Moaning 

G.  Precision 

H.  Timeliness 

I.  Training 

J.  Workload 


E 

F 
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.75 

.98 

.43 

.71 
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.91 

.18 

.41 

.94 
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.77 

.52 
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.82 

.57 

.30 

.75 

.54 
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.96 

.87 

.95 

.87 

.90 

Tabh  10.2-2.  Vartkai  Situation  Display  —  Corraiation  Matrix  of  Ratings 


Symbology  attributes 


A.  Usability  in  color 

B.  Cor>»picuousness  in  color 

C.  Usability  in  monochrome 

0.  Conspicuousness  in  monochrome 

E.  Location 

F.  Meaning 

G.  Precision 

H.  Timeliness 

I.  Tiaining 

J.  Workload 


Tabh  10.2-3.  Horizontal  Situation  Display  -  Corraiation  Matrix  of  Ratings 


Symbology  attributes 


A.  Usability  in  color 

B.  Conspicuousness  in  color 

C.  Usability  in  monochrome 
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G.  Precision 
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CONCLUSIONS  AND  FORMAT  REVISIONS 


The  basic  study  was  general  in  the  sense  that  formats  were  developed  and  tested  for 
the  HUD,  the  VSD,  the  HSD,  and  airplane  systems  formats  on  the  two  MPDs.  Some  of 
the  format  improvements  from  the  basic  study  were  implemented  in  the  threat 
warning  study.  In  that  second  study,  more  threat  information  was  added  to  the  HUD, 
VSD  and  HSD  formats  and  the  emphasis  in  testing  was  on  the  added  threat 
information.  Since  there  was  some  format  evolution  between  the  two  studies  and 
since  the  objective  and  methodology  of  the  two  studies  were  similar,  conclusions  and 
recommended  format  changes  are  combined  and  presented  in  this  final  section.  The 
revised  figures  for  the  HUD,  the  VSD,  and  the  HSD  formats  represent  composites.  As 
such,  they  include  examples  of  specific  symbology  which  may  not  all  appear 
simultaneously  in  actual  use. 

11. 1  General  Conclusions 

The  first  objective  was  to  determine  whether  these  formats  are  usable  by,,  and 
acceptable  to,  operational  military  fighter  and  attack  pilots.  Both  the  opinion  and 
performance  data  indicate  that  the  concept  of  pictorial  formats  and  the  particular 
formats  studied  were  usable  and  accepted  by  the  subject  pilots. 

The  second  objective  was  to  determine  whether  the  acceptability  and  usability  of 
these  formats  was  a  function  of  display  mode  -  color  fill,  color  line  or  monochrome  in 
the  basic  study,  and  color  or  monochrome  in  the  threat  warning  study.  In  both  studies, 
the  pilots  clearly  preferred  the  color  format  versions  and  their  performance  was 
slightly  better  with  color. 

The  firuJ  objective  was  to  identify  format  improvements.  From  pilot  opinion  and  weak 
spots  in  performance,  a  number  of  improvements  have  been  identified  and  are 
discussed  in  the  subsections  which  follow. 

This  is  not  to  say  that  the  work  on  pictorial  format  displays  is  completed  with  these 
revised  formats,  ^ach  specific  application  will  bring  its  own  set  of  requirements 
which  must  be  met,  some  with  more  empirical  study.  Since  crew  information  needs  in 


the  tactical  environment  may  change  considerably  with  different  mission  situations 
and  operating  conditions,  the  flexibility  offered  by  electronic  displays  and  information 
processing  should  be  fully  utilized  to  accommodate  changing  requirements.  Thus,  the 
particular  group  of  graphic  formats  evaluated  in  this  study  should  only  be  considered  a 
representative  subset,  subject  to  refinement  and  revision  as  needs  change. 

Also,  blending  of  pictorial  format  work  with  other  on-going  advanced  crew  station 
technology  will  provide  additional  opportunities  for  controlled  evolution  of  pictorial 
format  displays.  Activity  in  such  areas  as  flight  management,  mission  management, 
sensor  development  for  target  acquisition  and  flight  path  selection,  target 
classification,  and  weapon  delivery  ail  require  crew  station  integration  with  pictorial 
format  displays  to  help  achieve  the  transition  of  air  crews  from'  system  operators  to 
mission  managers,  as  discussed  in  Paragraph  1.1. 

Finally,  the  present  program  has  raised  a  number  of  new  issues.  Some  general  ideas 
are  presented  here,  and  those  which  are  format-specific  are  discussed  under  the 
specific  displays. 

Both  the  basic  study  and  the  threat  warning  study  were  aimed  at  simulating  and 
evaluating  candidate  concepts  for  pictorial  flight,  situation,  systems,  stores  and  threat 
warning  displays.  As  such,  the  displays  used  in  the  simulation  represented  a  first 
attempt  to  integrate  and  portray  a  wide  range  of  information  in  a  pictorial,  rather 
than  alphanumeric  or  strictly  analog  format.  In  order  to  progress  from  "concept" 
displays  used  in  laboratory  simulations,  to  displays  that  can  be  used  as  an  integral  part 
of  an  aircraft  Cockpit,  numerous  issues  must  be  resolved.  These  factors  range  from 
psychophysical,  perception-related  parameters  to  questions  of  information  structure 
and  priority,  and  include  such  questions  as  the  following: 

-  Display  field  of  view 

-  Optimum  size,  color,  duration,  brightness,  and  flash  rates  (where 
necessary)  for  symbology 

-  Optimum  display  size  and  placement 

Range  limits  for  terrain  and  threat  information 


Algorithms  for  threat  display  decisions  in  high  threat  density  situations, 
based  on  mode,  range  and  time  factors 

Pilot -selectable  declutter  options  and  format  changes  based  on  mission 
segments 

Role  of  voice  commands  in  entering  and  requesting  information,  and  of 
voice  annunciation  or  aural  warnings  for  threats  or  other  warning 
information 

11.2  Head-Up  Display 

The  HUD  display  format,  as  tested  in  the  basic  study,  was  not  a  unanimous  success. 
Pilots  tended  to  have  difficulty  with  it  initially,  and  even  after  a  significant  amount  of 
practice  this  format  would  have  to  be  considered  a  high  workload  contributor. 
Overall,  nearly  one-third  of  the  pilot  ratings  of  this  display  suggested  that  it  was  the 
same  or  more  difficult  to  use  than  information  in  conventional  aircraft.  The  usability 
of  the  HUD  received  a  lower  rating  than  any  of  the  formats  tested. 

Several  factors  iikeiy  contributed  to  this  result.  The  three-dimensional  flight  director 
pathway,  including  the  dynamics  of  flight  path  commands  and  "airplane"  handling 
qualities,  were  all  unfamiliar  to  the  evaluators.  The  concern  with  color  fill 
obscuration  of  the  outside  scene  during  VMC  was  the  major  source  of  lower  ratings. 
As  the  primary  flight  display,  and  as  the  only  source  of  this  type  of  information  during 
low  altitude  mission  segments,  the  HUD  would  be  expected  to  command  a  high 
percentage  of  the  pilot's  attention.  This  was  further  accentuated  by  the  degree  of 
precision  in  flight  control  elicited  by  the  flight  director  channel  during  manual  flight. 
The  dimensions,  sensitivity  and  dynamics  of  this  symbology  can  undoubtedly  be 
improved  upon.  Moreover,  the  precise  flight  path  control  called  for  during  particular 
segments  of  TF/TA  and  weapon  delivery  should  probably  be  relaxed  at  times  when 
requirements  are  less  severe. 

Nevertheless,  a  large  number  of  improvements  were  suggested  for  the  HUD  format 
and  have  been  incorporated  in  the  revision.  A  simple  steering  symbol  has  been  added. 
This  symbol  would  appear  any  time  that  the  entry  gate  of  the  pathway  is  at  the 
peripheral  limits  of  the  HUD  display,  and  would  provide  the  pilot  with  director 
information  to  smoothly  re-intercept  the  desired  pathway.  At  the  pilot's  option, 
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steering  symbol  dynamics  and  geometry  could  be  coupled  with  a  cursor  in  the  HSD  that 
allows  pilot  selecti  >1  of  the  re-interception  point.  This  has  the  further  advantage  of 
allowing  close-in  rrinual  threat  and  missile  avoidance  maneuvering  with  easy  return  to 
the  preferred  flight  route  at  any  desired  point. 

Indicators  have  been  added  for  air-to-air  and  surface-to-air  threats.  These  symbols 
would  appear  for  a  specified  period  whenever  the  onboard  threat  warning  system 
detects  pop-up  threats  not  previously  known  to  the  system,  and  would  serve  principally 
to  alert  the  pilot  to  new  threat  information  appearing  on  the  situation  displays. 

An  optional  vertical  velocity  indication  has  been  added,  primarily  to  assist  the  pilot 
with  precision  altitude  control  and  pitch  trimming  functions.  To  eliminate  any 
ambiguity  in  viewing  the  three-dimensional  flight  pathway,  particularly  during 
maneuvering  turns  where  the  channel  may  overlap  itself,  the  1000-foot  segment  lines 
have  been  eliminated  from  the  outside  and  used  only  on  the  inside  of  the  pathway. 

For  medium  and  hi<tl.  altitude  flight,  a  pilot  selectable  pitch  ladder  and  roll  index  are 
recommended.  This  could  be  accompanied  by  some  damping  of  the  flight  director 
channel  to  allow  greater  latitude  in  flight  path  control.  Abbreviated  pitch  and  roll 
indices  could  also  be  considered  for  low  altitude  segements  where  terrain  portrayed  in 
the  HUD  field-of-view  may  not  always  provide  a  reliable  horizontal  reference.  For 
transparency,  the  color  line  HUD  format  is  preferred  under  VMC  conditions,  but  may 
lack  the  information  needed  in  a  primary  flight  display  during  low-altitude  IMG 
operations* 

For  the  threat  warning  study,  the  transitional  flight  director  symbol,  and  the  threat 
alert  symbol  suggested  above  were  added  to  the  HUD  as  described  in  paragraph  8.1.1. 

In  the  threat  warning  study,  the  HUD  display  format  in  general,  and  the  threat  alerting 
symbology  in  particular,  were  fairly  well  accepted  by  the  pilots,  although  they  made 
many  suggestions  for  specific  coding  changes.  Usability  of  most  of  the  threat 
symbology  was  rated  moderately  good  to  very  good;  the  only  exception  was  the  threat 
azimuth  vector,  which  was  difficult  to  see  under  some  conditions.  Many  of  the  pilot 
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suggestions  for  change  were  requests  for  more  (or  more  differentiated)  threat-related 
information  to  be  presented  on  the  HUD.  Specific  suggestions,  and  their  impact  on  the 
revised  formats  are  described  below. 

The  only  HUD  symbology  that  received  a  usability  rating  poorer  than  "neutral"  was  the 
transitional  flight  director.  As  discussed  previously,  this  symbology  was  included  for 
the  first  time  in  the  threat  warning  study,  and  its  implementation  was  incomplete. 
Pilot  comments  supported  the  concept  of  a  flight  director,  but  not  its  execution  in  this 
simulation. 

The  HUD  alert  symbology  consisted  of  an  air-to-air  alert  symbol  (shaped  like  a  fighter) 
or  a  surface-to-air  symbol  (a  missile)  that  appeared  solid  for  six  seconds  for  search¬ 
mode  threats;  flashed  for  six  seconds  for  track-mode  threats;  or  flashed  for  launch 
duration,  accompanied  by  a  time-to-missile-impact  readout  and  threat  azimuth  vector, 
for  launch-mode  threats.  Most  of  the  pilots  considered  the  HUD  threat  symbology 
adequate  for  alerting  them  to  change  in  the  threat  environment;  and  also  rated  it 
moderately-  to  very  usable,  especially  in  color.  Half  the  pilot;,  however,  suggested 
that  a  separate  alert  symbol  be  used  for  AAA's,  to  distinguish  them  from  SAMs. 
Figure  11.2-1  shows  the  revised  HUD  format,  which  incorporates  an  AAA  alert  symbol, 
a  gun-type  silhouette  at  bottom  right. 

Some  pilots  also  commented  that  it  was  sometimes  difficult  to  distinguish  between 
track-  and  launch-mode  alert  symbology,  which  differed  only  in  duration  and  in  the 
presence  or  absence  of  the  time-to-impact  readout  and  threat  azimuth  vector.  In  the 
revised  HUD  format,  search  mode  threats  would  be  represented  by  solid  (non-flashing) 
yellow  symbols  (SAM,  AAA  or  A/A,  as  appropriate);  track-mode  threats  by  solid  red 
symbols;  and  launch  mode  threats  by  flashing  red  symbols.  This  coding  would  make  the 
distinction  between  threat  modes  more  explicit.  In  addition,  since  threat  azimuth 
symbology  would  not  be  tied  directly  to  launch  mode  threats,  it  could  be  added,  as 
several  pilots  suggested,  to  track  mode  threat  alerts,  or  eliminated  where 
inappropriate.  The  tirne-to-missile-impact  readout  could  be  deleted  for  AAA's,  for 
example,  to  increase  the  distinction  between  them  and  SAM's,  as  suggested  by  two 
pilots.  Table  11.2-1  summarizes  the  revised  HUD  threat  alert  coding. 


Figure  1 1.2- 1.  Revised  Head-Up  Display  (HUD)  —  Composite 


Some  pilots  requested  more  continuous  threat  information  (for  search  and  track  mode 
threats)  in  addition  to  the  alerts  for  new  threats.  One  possible  way  of  providing  longer 
and  more  inclusive  information  about  the  overall  "active  threat  situation"  is  shown  in 
Figure  11.2-1.  Threat  alert  symbology  similar  to  that  in  the  threat  warning  study,  but 
with  the  modifications  described  above,  would  continue  to  appear  below  the  aircraft 
symboi  for  all  new  or  upgraded  mode  threats,  to  alert  the  pilot  to  changes  in  the 
threat  environment.  In  addition,  similar  symbology,  but  reduced  in  size,  would  appear 
for  all  currently  active  threats  in  a  row  below  the  threat  alert  symbology,  at  the 
bottom  of  the  screen.  The  symbol  for  a  given  type  of  threat  (A/A,  SAM  or  AAA) 
would  always  appear,  if  a  threat  of  that  type  was  active,  in  the  same  relative  positions 
left,  center,  or  right.  The  "currently  active”  threat  symbols  would  be  coded  in  the 
same  way  as  the  "threat  eiert"  symbology:  yellow,  red,  or  flashing  red  for  search, 
track,  or  launch  modes.  If  more  than  one  threat  of  a  given  type  were  active,  the 
number  of  those  active  could  be  given  below  the  appropriate  symbol,  which  would  be 
coded  to  represent  the  most  severe  mode  of  the  threats  of  that  type.  Figure  11.2-1 


Table  1 1.2-1.  Proposed  Threat  Information  Coding  for  Head-Up  Display  Format 


Information 

Scooe  or  restrictions 

Levels 

Coding 

Threat  status 

- 

•  "Alert" 

•  Large  threat  symbol  just  below 
"own"  aircraft  symbol,  6  seconds 

■ 

•  "Active" 

•  Smaller  threat  symbols  below 
alert  threat  symbol,  duration 
of  active  mode 

Threat  type 

•  Alert  or  active  threats 

•  Air-to-air 

•  Fighter  symbol 

•  SAM 

•  Missile  symbol 

•  AAA 

•  Gun  symbol 

Threat  mode 

•  Alert  or  Active  threats 

•  Search 

•  Yellow  threat  symbol 

•  A/A,  SAM  or  AAA 

Highest  priority  threat 
of  given  status,  type 

•  Track 

•  Launch 

•  Red  threat  symbol 

•  Flashing  red  threat  symbol 

Countermeasures  status 

•  Alert  or  Active  threats 

•  Effective 

•  Yellow  circle 

•  A/A,  SAM  or  AAA 

Most  revere  status  for 
given  status,  type 

•  Track  or  Launch  mode 
only 

•  Not  effective 
or  depleted 

•  Red  circle 

Missile  azimuth 

•  Alert  threat 

0-  to  360-deg 

Vector  to  missile  azimuth 

.  A/A  or  SAM 

•  Launch  mode  only 

■ 

Missile  "Time-to-lmpact" 

•  Alert  threat 

•  A/A  or  SAM 

•  Launch  mode  only 

0  to  x  secs 

Boxed  readout  of  time  to  missile 
impact 

Number  of  threats 

•  Active  threats 

•  A/A,  SAM  or  AAA 

■ 

2  to  x 

Default  is  1;  pilot's  option  to  select 
"all  active  threats"  number,  or  restrict 
to  track  and  launch  mode  total 

also  shows  possible  coding  for  the  countermeasures  status  of  "active"  or  "alert" 
threats,  using  a  yellow  (effective)  or  red  (not  effective)  circle  around  the  appropriate 
symbol  (for  track  or  launch  modes  threats,  analogous  to  the  coding  used  on  the 
Vertical  or  Horizontal  Situation  Displays).  Four  pilots  suggested  that  countermeasures 
information  be  added  to  the  HUD. 


The  additional  symbology  for  "active"  threats  and  countermeasures  status  information 
outlined  above  and  shown  in  Figure  11.2-1  is  only  one  possible  approach.  The 
feasibility  of  adding  such  information  as  well  as  its  size,  location.,  range  limits,  and 
availability  a  pilot-selectable  mode,  for  example)  would  need  to  be  investigated  in 
further  research. 

Two  other  changes  have  been  incorporated  into  the  revised  HUD  symbology  in  response 
to  pilot  comments:  the  threat  azimuth  has  been  made  longer  and  thicker,  and  the 
time-to-missile-impact  readout  has  been  enlarged  to  make  it  more  conspicuous. 

The  revisions  discussed  above  for  the  HUD  are  related  to  threat-warning  symbology. 
As  in  the  basic  study,  some  of  the  pilots  suggested  that  a  pitch  ladder  and  better  flight 
direction  information  be  added  to  the  HUD.  Changes  in  the  sensitivity  of  the  flight 
channel  are  appropriate  for  different  mission  segments  and  requirements.  The 
addition  of  an  abbreviated  pitch  ladder,  pilot-adjustable  sensitivity  for  the  flight 
channel  symbology,  end  a  fully  implemented  flight  director  for  recapturing  the  flight 
channel  when  necessary,  would  improve  the  usability  and  pilot  acceptability  of  the 
HUD  format. 

11.3  Vertical  Situation  Display 

The  low-altitude  VSD  format  was  a  true  situation  display  that  portrayed,  in  aircraft 
stabilized  form,  the  interaction  of  one's  own  aircraft  with  terrain,  threats,  targets, 
cultural  features,  and  future  intended  flight  path.  At  high  altitude  the  VSD  reverted 
to  a  standard,  ground-stabilized  electronic  attitude  director  indicator  (EADI)  format. 
Generally,  the  unique  features  of  the  low  altitude  VSD  were  well  accepted;  it  provided 
a  concise  overview  of  the  mission  situation  with  a  minimum  requirement  to  process 
and  integrate  information  from  different  sources.  Overall,  three-fourths  of  the  pilot 
evaluators  rated  this  concept  as  easier  to  use  than  conventional  aircraft  information. 

Much  of  the  criticism  of  the  vertical  situation  display  in  the  basic  study  followed  the 
form,  "I  liked  it,  particularly  for  threat  display,  but  otherwise  it  was  redundant."  In 
part,  this  stemmed  from  lack  of  understanding  or  acceptance  of  the  distinction 
between  flight  and  situation  display.  The  low  altitude  VSD  was  intended  to  be  a 
situation  display  showing  more  about  the  threat  and  terrain  environment  and  less  about 
the  mechanics  of  flight.  The  latter  information  was  contained  in  the  HUD. 


US 


If  the  HUD  had  been  optimized  and  if  the  vertical  situation  display  had  remained  a 
situation  display  at  high  altitude,  the  pilots  would  likely  have  been  quite  satisfied  with 
the  VSD.  The  low  altitude  VSD  changes  are  shown  in  Figure  11.3-1.  The  ground  grid 
should  have  meaningful  linear  dimensions  (1  mile  squares)  and  be  fixed  at  the  nadir  or 
point  directly  below  the  aircraft.  This  would  help  in  determination  of  range,  to  terrain 
and  threats  and  also  give  a  coarse  indication  of  aircraft  altitude.  There  was  some 
feeling  that  a  heading  tape  or  compass  rose  was  needed  somewhere  as  an  azimuth 
reference  for  terrain,  threats  and  targets  and  one  suggestion  was  the  VSD.  That  is 
probably  appropriate  since  it  has  a  wider  field  of  view  than  the  HUD.  If  necessary,  a 
more  complete  compass  rose  could  be  added  to  the  HSD. 


Figure  11.3- 1.  Revised  Low  Altitude  Vertical  Situation  Display  (VSD)  -  Composite 


At  high  altitude,  the  VSD  should  not  dramatically  change  character  from  an  aircraft- 
stabilized  situation  display  to  a  ground-stabilized  attitude  director  display.  Even  if 
the  aircraft  is  above  terrain,  the  threat  Information  may  still  apply.  The  ground  grid 
could  be  retained,  with  each  square  representing  five  or  ten  miles  to  reduce  clutter. 
The  VSD  could  still  be  a  back-up  location  for  the  HUD  format  in  case  of  HUD  system 
failure. 

With  one  exception  (altitude  terrain  coding),  the  color  version  of  the  VSD  received 
ratings  of  slightly  good,  or  better,  in  the  threat  warning  study.  However,  in  general, 
the  VSD  was  the  lowest  rated  display  and  the  least  used  source  of  threat  information. 
In  part,  this  result  was  an  artifact  of  the  response  measures,  and  the  relatively  narrow 
field  of  view  of  the  VSD,  combined  with  an  anomaly  in  the  threat  mode  coding  for  the 
VSD.  Moreover,  it  was  likely  influenced  by  the  design  of  the  VSD  format  as  a 
“situational"  display,  rather  than  one  intended  to  give  discrete  threat  data  uniquely 
tailored  for  response  purposes. 

In  the  threat  warning  study,  the  pilots  were  required  to  give  the,  type  and  mode  of  all 
threats  flagged  by  a  HUD  threat  alert  symbol.  Sometimes  such  threats  were  out  of 
the  field  of  view  of  the  VSD,  or  hidden  behind  a  mountain  or  another  threat. 
Information  for  such  threats  could  be  reliably  obtained  only  be  referring  to  the  HSD. 
This  was  especially  true  for  threats  in  track  mode.  Once  they  went  out  of  the  field  of 
view  of  the  VSD,  their  lethality  envelope,  tractor  beam,  and  the  concommitant 
countermeasures  status  coding  circle  disappeared;  this  was  not  true  for  the  HSD.  An 
important  revision  sroposed  for  the  VSD  is  that  the  tractor  beam  and  countermeasures 
information  for  track  mode  threats  remain  on  the  VSD  even  if  the  threat  goes  outside, 
the  field  of  view  (with  the  caveat  that  some  range  limits  might  have  to  be  enforced 
for  all  three  displays,  especially  in  a  high  density  threat  environment). 

Another  difficulty  with  the  VSD  (one  it  shared  with  the  HSD)  was  the  "jewel”  light 
symbology  used  to  replace  the  lethality  envelopes  of  threats  in  launch  mode.  Pilot 
comments  (and  performance  data)  indicated  that  the  jewel  light  was  easily  confused 
with  the  AAA  threat  lethality  envelope.  Moreover,  once  the  jewel  light  had  replaced 
the  threat  lethality  envelope,  there  was  only  minimal  information  as  to  the  type  of 
threat  that  was  in  launch  mode  (i.e.,  the  presence  or  absence  of  the  missile  position 
symbol  on  the  HSD,  and  the  difference  in  time-to-missiie-impact  readout  on  the  HUD). 


Table  1 1.3- 1.  Proposed  Threat  Mode  Coding  for  Low  Altitude  Vertical  Situation  Display 
and  Low  Altitude  Horizontal  Situation  Display  Formats 
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*  An  increase  in  the  selected  HSD  range  would  bring  the 
threat  site  within  the  HSD  display  range 


Finally,  there  was  a  more  general  problem  with  the  way  the  jewel  light  was 
implemented  in  the  threat  warning  simulation.  Although  the  substitution  of  the  jewel 
light  was  initially  intended  to  indicate  that  a  threat  lethality  envelope  had  been 
penetrated,  ther*»  was  also  a  requirement  that  the  jewel  light  remain  as  long  as  the 
threat  was  in  launch  mode.  Thus,  threat  mode  status  and  lethality  envelope 
penetration  were  not  clearly  differentiated.  To  keep  the  coding  and  briefing 
straightforward,  the  jewel  light  was  defined  as  a  part  of  the  code  for  launch  mode 
status  rather  than  lethality  envelope  penetration.  One  possible  solution  for  the 
drawbacks  associated  with  jewel  light  as  it  was  used  in  the  threat  warning  study  is 
shown  in  the  revised  VSD  and  HSD  formats,  Figures  11.3-1  and  11.4-1,  and  the 
proposed  threat  coding  for  the  two  formats,  Table  11.3-1. 


Figure  11.3-1  shows  the  jewel  light  replaced  by  a  "threat  site  symbol"  similar  to  the 
threat  symbology  used  in  the  HUD.  This  symbology,  with  a  unique  symbol  for  AAA's, 
SAM's  and  A/A  threats  would  allow  threat  type  to  be  identified  even  after  the  threat 
lethality  envelope  disappeared.  In  addition,  Table  11.3-1  shows  how  all  combinations 
of  threat  mode  (search,  track  and  launch),  threat  envelope  penetration  (inside  or 
outside  envelope"  and  VSD  field  of  view  (inside  or  outside  field  of  view)  could  be 
represented. 

With  this  coding,  the  appropriate  threat  site  symbol  would  replace  a  lethality  envelope 
only  if  the  envelope  had  been  penetrated.  The  threat  site  symbol  would  then  reflect 
the  current  mode  of  that  threat:  the  symbol  would  be  yellow,  red,  or  flashing  red  for 
threats  in  search,  track,  or  launch  mode  (with  the  latter  modes  more  probable). 
Threat  site  symbols  would  appear  only  for  threats  within  the  VSD  field  of  view;  for 
track  and  launch  mode  threats  outside  the  field  of  view,  the  appropriate  tractor  beam 
(*oiid  or  flashing  to  correct  azimuth)  and  countermeasures  status  circle  (yellow  or  red) 
would  remain.  The  width  of  the  tractor  beam  would  be  increased  to  make  it  more 
conspicuous,  and  the  optimum  flash  rate  would  be  determined.  Similar  coding  would 
be  used  on  the  HSD.  This  proposed  coding  is  only  one  possible  approach,  but  it  has  the 
dual  advantages  of  presenting  and  distinguishing  threat  envelope  penetration  and 
threat  mode  information,  as  well  as  increasing  the  consistency  and  commonality  of 
coding  across  displays.  This  would  have  the  added  result  of  making  the  VSD  a  much 
more  reliable  and  complete  source  of  threat  information,  increasing  its  usability  as  a 
true  situational  display. 

11.4  Air-to-Surface  Horizontal  Situation  Display 

The  basic  HSD  is  an  electronic  moving-map  display.  It  contains  some  unique  features 
which  should  be  retained.  The  idea  of  showing  threat  envelope  and  mountain  sizes  at 
current  altitude  is  good  and  would  probably  be  even  more  valuable  with  denser  threats 
and  more  manual  evasive  maneuvers  than  were  simulated  here.  The  target  inset  was 
well  accepted  by  most  of  the  pilots. 

Several  optional  features  should  be  added.  If  numeric  fuel-range  is  important,  then  its 
numeric  value  should  be  added  to  the  fuel-range  rings.  The  selectable  time  and 
distance  boxes  should  have  a  "direct"  option  as  well  as  "via  flight  plan".  Other 


important  ground  features  such  as  prime  and  alternate  landing  fields  and  radio  aids  to 
navigators  should  be  considered  as  pilot  options.  An  offset  option  should  be  added  to 
put  the  airplane  symbol  at  the  bottom  of  the  display.  This  would,  for  a  given  scale 
selection,  almost  double  the  look-ahead  range.  A  pilot-selectable  cursor  is  suggested 
that  would  allow  the  pilot  to  make  minor  course  changes  in  near  real-time,  and  would 
facilitate  return  to  the  preplanned  route  following  manual  maneuvering. 

An  optional  trend  vector  should  be  added.  This  would  give  30,  60  and  90-second 
predictions  based  on  current  configuration  and  speed.  In  other  applications,  this 
feature  has  proven  to  be  a  valuable  aid  to  manual  flight.  Display  range  (scale)  should 
be  stated  as  distance  from  the  airplane  symbol  to  the  top  of  the  display. 

In  the  threat  warning  study,  the  horizontal  situation  display  received  the  highest 
usability  ratings  of  the  three  displays.  This  result  may  have  been  partly  an  artifact  of 
the  response  structure  and  some  anomalies  in  the  VSD  threat  coding,  as  discussed 
earlier.  The  revisions  to  the  H5D  air-to- surface  format,  shown  in  Figure  11.4-1  and 
described  in  Table  11.3-1,  are  analogous  to  those  made  on  the  VSD.  The  primary 
proposed  change  is  the  .substitution  of  a  threat  site  symbol  (which  also  codes  threat 
mode  status)  for  threats  within  the  USD  display  range  whose  lethality  envelopes  have 
been  penetrated.  As  with  the  VSD,  the  tractor  beam  (for  track  or  launch  modes)  would 
be  made  thicker  and  more  conspicuous,  and  it  would  Hash  for  launch  mode  to  increase 
consistency  across  displays. 


Because  the  plan  view  of  the  HSD  contains  no  information  on  the  height  of  the 
lethality  envelopes,  the  pilots  had  more  difficulty  distinguishing  SAM's  from  AAA's, 
and  threats  (especially  AAA's  and  jewel  lights)  from  terrain.  One  reason  for  this  was  a 
less  discriminable  difference  between  the  red  (of  threats)  and  the  brown  (of  above¬ 
altitude  terrain)  on  the  CRT  used  in  this  simulation  for  the  HSD.  But  this  may  be  only 
one  example  of  the  more  general  problem  of  distinguishing  threats  from  terrain  when 
more  realistic  depictions  are  used  (so  that  threats  and  terrain  cannot  be  classified  by 
their  distinctive  regular  shapes),  or  when  a  more  dense  threat  and  mountain 
environment  occurs.  Although  it  was  not  incorporated  in  the  revised  HSD  format 
shown  in  Figure  1 1.4-1,  one  possible  solution  would  be  to  use  a  more  distinctive  coding, 
such  as  a  striped  (stroke  or. partial  fill  raster)  format  for  terrain.  This  would  increase 
the  discriminability  of  the  two  types  of  information,  and  would  also  allow  the  terrain 


information  to  be  presented  in  a  high  density  environment  without  occluding  important 
threat  information.  This  is  only  an  attempt  to  define  a  possible  approach  to  a  solution, 
since  no  work  has  been  done  to  test  the  effects  of  this  coding  in  the  laboratory. 

One  final  pilot  suggestion  should  be  noted:  three  pilots  wanted  some  indication  on  the 
HSD  about  which  threat  was  associated  with  the  threat  alert  symbol  currently  on  the 
HUD.  This  was  undoubtedly  influenced  by  the  requirement  to  identify  the  clock 
position  of  the  threat  for  a  given  HUD  alert,  but  this  request  could  be  easily 
accommodated  by  changing  the  color  of  the  entire  threat  envelope  for  the  duration  of 
the  HUD  alert  symbol  (e.g.,  magenta  threat  envelope  for  six  seconds).  The  feasibility 
of  this  suggestion  would  depend  on  threat  density  and  the  timing  of  threat  mode 
changes. 

UJ  Air-to-Air  Horizontal  Situation  Display 

The  air-to-air  format  of  the  HSD  was  well  received.  The  revised  format,  shown  in 
Figure  11.5-1,  reflects  two  major  changes.  First,  in  keeping  with  the  changes  made  to 
the  air-to-surface  HSD,  and  the  VSD,  a  "threat  site"  symbol  replaces  the  lethality 
envelope  of  the  threat  (the  sector  shape)  once  it  has  been  penetrated. 

Second,  the  location  of  the  threat  number,  type,  and  altitude  difference  information 
has  been  changed.  Some  pilots  noted  that,  for  some  ranges,  this  information  was  out 
of  the  HSD  field  of  view.  To  provide  more  timely  information,  these  data  have  been 
moved  inside  the  threat  envelopes.  Threat  number  might  also  be  coded  pictorially,  as 
shown  in  Figure  1 1.5-1.  This  coding  gives  a  clearer  picture  of  the  relative  position  of 
formation  aircraft,  transitions  more  easily  when  a  change  in  formation  occurs,  and 
reAices  the  alphanumeric  readouts  to  two  numbers  that  can  fit  in  the  body  of  the 
throat  envelope  (even  if  the  envelope  rotates)  under  ail  usable  scale  conditions. 

11*6  Multipurpose  Displays 

The  status  and  advisory  formats,  which  were  shown  on  the  two  multipurpose  displays, 
were  generally  welt  received.  Some  of  them  do  not  warrant  revision  at  this  time. 
Others  require  extensive  work. 


Figure  1 1.5 ■  1.  Revised  Air  to- Air  Horizontal  Situation  Display  (HSD)  —  Composite 

11.6.1  Engine  Formats 

The  pilots  wanted  two  things  from  engine  displays.  They  required  good  thrust  setting 
information  and  they  were  concerned  about  engine  health.  The  test  version  of  the 
engine  format  covered  both  of  these  concerns  -  but  not  weiL  There  was  not  sufficient 
resolution  in  the  thrust  parameters  (NI,  N2,  fuel  flow)  to  set  thrust  level.  The  engine 
health  parameters  (turbine  inlet  temperatures,  exhaust  gas  temperature)  were  lost 
among  the  other  display  elements;  engine  oil  quantity  and  pressure  were  not  shown. 
The  engine  status  display  was  criticized  for  being  too  cluttered. 

The  design  response  to  these  criticisms  is  shown  in  Figure  11.6.1-1.  Turbine  inlet 
temperature  and  compressor  speed  have  been  eliminated.  Fuel  flow  has  been 
simplified  and  "bubble"  movement  removed.  If  numeric  fuel  flow  is  required,  it  could 
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Figure  11.6.1-1.  Engine  Status  Format 


be  added  at  the  ends  of  the  fuel  flow  arrows.  Characters  should  be  the  same  size  as 
the  letters  on  this  format.  A  composite  thrust  parameter  is  suggested  and  displayed  as 
shown  in  a  tape- like  display  with  desired  thrust  shown,  derived  from  autopilot  or 
manually  set  desired  airspeed.  Actual  thrust  is  shown  along  with  throttle  position. 
This  way,  the  thrust  goal  is  known  and  displayed.  The  pilot  can  set  his  throttles 
quickly  to  match  that  goal  and  attend  to  other  duties  while  engine  thrust  comes  up  (or 
down)  to  match  the  setting  or  goal.  The  procedure  obviates  the  continual  monitoring 
and  successive  approximation  frequently  required  with  current  engine  display  systems. 
The  engine  advisory  formats  were  successful.  They  cued  the  pilots  though  the  steps 
necessary  to  deal  with  engine  problems.  No  change  is  suggested  to  the,  existing 
formats.  An  engine  health  format  should  be  added  to  display  the  health  parameters  of 
concern. 

11.6^2  Stores  Formats 

The  stores  status  and  stores  program  formats  were  very  successful  -  they  were  well 
accepted  and  pilots  performed  well  with  them.  Neither  the  status  nor  the  program 
formats  require  revision  ut  this  stage.  Both  would  be  tailored  for  specific 
applications.  Pilot  errors  in  dealing  with  the  hung  bomb  problem  indicate  that  stores 
problems  should  bring  up  either  verbal  or  pictorial  checklists  like  those  used  here  for 
engine  and  electrical  problems. 

11-6.3  Electrical  Formats 

The  electrical  status  format  requires  only  modest  change.  The  distinction  between 
permanent  and  temporary  failure  of  an  electrical  system  element  was  subtle  -  filled 
vs.  open  X.  If  the  distinction  is  important,  it  should  be  more  clearly  indicated.  Figure 
11.6.3-1  is  the  modified  electrical  status  format,  showing  a  temporary  failure  of  the 
left  generator.  A  permanently  failed  element  would  have  an  X  through  it,  as  well. 

The  electrical  advisory  format  was  an  example  of  a  pictorial  checklist  showing 
required  actions.  Some  pilots  would  have  preferred  verbal  checklists  and  were  critical 
of  the  cartoons.  If  verbal  checklists  are  used,  they  should  be  interactive  and  have  a 
recall  feature.  In  this  way,  completed  checklist  items  would  be  automatically 
indicated  and  actions  not  completed  would  be  available  for  later  recall. 
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Figure  11.6.3-1.  Electrical  Status  Format 


If  pictorial  checklists  are  used,  they  should  be  simple,  clear  and  have  a  fairly  uniform 
format  across  systems.  Required  actions  should  be  clearly  separated,  perhaps  to  the 
point  of  having  separate  pictures  for  each  action.  This  would  have  precluded  the 
relatively  common  error  of  forgetting  to  turn  the  generator  back  on  after  a  temporary 
failure.  In  pictorial  checklists,  condition  and  action  should  be  clear.  Showing  cockpit 
location  of  switches  is  not  as  important  and  could  be  excluded. 


11.6.4  Hydraulic  Formats 

The  hydraulic  status  format  was  well  received  and  easily  understood.  However,  a 
number  of  pilots  indicated  the  need  for  numeric  information.  Pending  the  sort  of 
analysis  that  accompanies  a  specific  application,  no  revision  is  recommended. 

The  hydraulic  advisory  format  showed  results  of  a  system  failure  but  did  not  indicate 
any  action  required.  The  pilots  read  the  display  quickly  and  accurately.  No  revision  is 
recommended. 

11.6.3  Fuel  System  Format 

The  fuel  status  format  was  judged  to  be  good,  but  pilots  had  to  refer  to  a  conventional 
gauge  to  report  amount  of  remaining  fuel.  Figure  11.6.5-1  shows  a  version  which  adds 
numeric  fuel  quantity  to  the  format.  In  some  aircraft,  fuel  flow  is  used  as  an 
indication  of  thrust.  A  number  of  pilots  requested  display  of  fuel  flow.  This  has  been 
discussed  in  the  revised  engine  status  format. 


Figure  11.6.5-1.  Fuel  Status  Format 
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APPENDIX  A 


PILOTS'  QUESTIONNAIRE  -  BASIC  STUDY 

This  appendix  is  a  copy  of  the  questionnaire  completed  by  the  pilots  in  the  basic  study. 
As  described  in  paragraph  5.4  of  the  main  body  of  this  report,  the  first  part  of  the 
questionnaire  was  analyzed  by  the  method  of  magnitude  estimation.  The  numbers 
written  in  under  "Color  Fill"  and  "Monochrome"  are  the  geometric  mean  responses. 
Questions  marked  with  asterisks  had  significant  differences  (p  <  .05)  by  Friedman 
two-way  analysis  of  variance. 

In  the  second  part  of  the  questionnaire,  pilots  were  asked  to  compete  these  formats 
with  conventional  aircraft  displays.  The  boxes  in  that  part  of  this  appendix  contain 
the  number  of  pilots  making  that  response  and  the  asterisks  indicate  the  median 
response. 
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This  part  of  the  questionnaire  Is  designed  to  obtain  your  opinions  regarding  the  relative  usability 
of  the  three  presentation  modes  for  the  pictorial  formats:  monochromatic,  color  line,  and  color  fl 
Please  Indicate  the  jsablllty  of  presentation  modes  by  placing  a  number  on  each  line  under  the 
display  mode  column  headings  adjacent  to  each  question.  To  give  you  a  standard  reference  for  your 
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Thb  part  of  die  questionnaire  i*  designed  to  obtain  your  opinion  with  regard  to  tha  utability  of  the  pictorial  display  format!  that  ware  pratantad  in  the  simulated 
mission.  On  each  of  the  following  scales,  you  are  requested  to  indicate  the  relative  usability  of  a  specific  Pictorial  format  by  comparing  it  to  comparable  inf  or- 
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APPENDIX  B 


SYNOPSIS  OF  TWENTY  RESPONSES  TO  EACH  OF  NINETEEN  GENERAL  QUESTIONS 

BASIC  STUDY 

It  was  hoped  that  providing  each  pilot  a  list  of  general  questions  and  a  tape  recorder 
would  elicit  ideas  not  otherwise  available  -  a  sort  of  directed  free  association.  The 
technique  worked  well.  Transcripts  of  the  tapes  from  individual  pilots  were 
recordered  by  question  and  are  summarized  in  the  pages1  following.  The  richness  of  the 
raw  data  made  summarizing  difficult  but  an  attempt  was  made  to  represent  all  the 
ideas  presented.  Both  convergences  and  divergences  appear.  There  were  a  few 
comments  on  the  simulation  as  opposed  to  the  formats  under  evaluation;  these  were 
excluded  from  the  summaries. 
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Question  1.  What  is  your  general  feeling  with  regard  to  the  use  of  CRT  displays  in  the 
cockpit? 

Good  (19  pilots). 

Systems  displays  especially  good  (5  pilots). 

Some  too  complicated  (4  pilots). 

Make  thrust  setting  information  better  (2  pilots). 

VSD  and  HSD  don't  give  quick  information  (1  pilot). 

Quick  and  easy  crosscheck  (1  pilot). 

Color  fill  best  (1  pilot). 

Question  2.  What  is  your  general  opinion  of  the  low  altitude  Vertical  Situation 
Display? 

Good,  particularly  threat  display  (IS  pilots). 

Good  for  situational  awareness  (2  pilots). 

Used  it  mostly  for  threats  (5  pilots). 

Ribbon  questionable  (3  pilots). 

Put  threat  warning  in  HUD,  as  well  (3  pilots). 

Perspective  confusing  (2  pilots). 

Want  better  steering  and  attitude  information  (2  pilots). 

Tie  to  radar  for  real  terrain  (2  pilots). 

Monochrome  best  (1  pilot). 

Color  line  best  (1  pilot). 

Color  fill  best  (1  pilot). 

Question  3.  Does  the  HUD  provide  sufficient  information  .for  flight  path  control 
during  low  altitude  flight?  If  not,  what  would  you  add? 

Yes,  the  HUD  provides  sufficient  information  for  flight  path  control  in  low 
altitude  flight  (8  pilots). 

No,  it  doesn't  (12  pilots). 

Add  better  flight  director  or  steering  information  (10  pilots). 
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Pathway  difficult  to  return  to  from  extreme  deviations  (7  pilots). 

Add  clear  attitude  information  (6  pilots). 

Get  rid  of  background  fill,  especially  in  VMC  (5  pilots). 

Add  threat  indication  (5  pilots). 

Add  vertical  velocity  (5  pilots). 

Heading,  airspeed  and  altitude  boxes  should  be  moved  in  (2  pilots). 

Add  forward-looking  sensor  overlay  (2  pilots). 

Add  pitch  ladder  (2  pilots). 

Too  cluttered  (2  pilots). 

Add  angle-of-attack  (1  pilot). 

Ridge  lines  too  cluttered  (1  pilot). 

Add  low  altitude  alert  (1  pilot). 

Use  velocity  vector  (1  pilot). 

Prefer  standard  ADI  (1  pilot). 

Question  ».  In  this  simulation,  the  HUD  presentation  was  airplane  stabilized  and  the 
Vertical  Situation  Display  was  ground  stabilized.  Is  this  acceptable?  If  not,  what 
would  you  change? 

Acceptable  (19  pilots). 

Not  acceptable  (i  pilot). 

Add  attitude  to  HUD  and  better  steering  display  on  VSD  (1  pilot). 

Establish  back-up  director  mode  for  VSD  if  HUD  fails  (1  pilot). 

Question  5.  What  do  you  think  of  the  terrain  shading  on  the  low  altitude  color  fill 
HUD  format? 

Fine  as  it  was  (8  pilots). 

Alright  for  IMC,  but  clear  up  for  VMC  (6  pilots). 

Masked  too  much,  color  line  better  (5  pilots). 

Attitude  and  steering  not  well  displayed  (2  pilots). 


Add  range  information  to  the  mountains  (1  pilot). 

Need  declutter  capability  (1  pilot). 

Color  fill  excellent  on  VSD  and  HSD  (1  pilot). 

Terrain  avoidance  information  is  better  on  the  A6  (1  pilot). 

Question  6.  Did  you  have  difficulty  interpreting  HUD  flight  path  information  due  to 
the  lag  between  the  airplane  symbol  and  the  pathway-in-the-sky  when  flying  with 
autopilot  engaged? 

No  difficulty  on  autopilot  (13  pilots). 

Problems  in  manual  flight  (6  pilots). 

Turns  hard  to  interpret  (3  pilots). 

Experience  may  help  (3  pilots). 

Monochrome  hard  to  interpret  (2  pilots). 

Hard  to  get  back  to  course  (1  pilot). 

Too  much  information  when  in  autopilot  (1  pilot). 

A6  format  better  (1  pilot). 

Question  7.  Did  you  ever  refer  to  the  electromechanical  standby  instruments  in  the 
cockpit?  If  so,  about  how  many  times  and  for  what  purpose? 

Rarely  (6  pilots). 

Sometimes,  particularly  on  early  flights  (5  pilots). 

Yes,  but  more  out  of  habit  (5  pilots). 

Yes,  primarily  for  cross  check  (3  pilots). 

Altimeter  (8  pilots). 

Used  vertical  speed  indicator  (5  pilots). 

Mach/air  speed  (5  pilots). 

Needed  analog  heading  information  (2  pilots). 

Looked  for  attitude  because  it  wasn't  clear  on  the  HUD  (1  pilot). 

Looked  for  angle  of  attack  (1  pilot). 

Would  like  to  see  Navy  format  angle  of  attack  on  HUD  (i  pilot). 


Note:  Despite  all  the  emphasis  here  on  pictorial  formats,  heading,  airspeed  and 
altitude  were  presented  digitally.  Pilots  used  the  standby  instruments  for  analog 
presentations  of  those  parameters. 

Question  8.  Did  the  flight  path  channel  ever  completely  disappear  from  the  HUD?  If 
so,  did  you  find  it  difficult  to  recapture  proper  position  with  respect  to  the  flight 
path? 

Note:  When  the  path  disappeared  from  the  HUD,  it  was  due  to  extreme  deviation  from 
the  flight  path.  In  that  case,  the  entry  gate  remained,  pegged  at  the  display  periphery. 

Yes,  it  did  but  recovery  was  not  difficult  (5  pilots). 

Yes,  and  add  steering  symbology  to  get  back  (5  pilots). 

Yes,  HSD  helped  recovery  (3  pilots). 

No,  but  steering  cues  would  help  (4  pilots). 

No,  it  did  not  disappear  (3  pilots). 

Question  9.  Was  the  blinking  velocity  vector  on  the  HUD  a  useful  indication  of 
abnormal  and  emergency  conditions  in  the  monochromatic  presentation? 

Yes,  it  was  fine  (11  pilots). 

No,  easy  to  miss  (4  pilots). 

Other  indications  (master  caution,  fire  light  or  blinking, display, selector)  were 
more  effective  (5  pilots). 

Question  10.  Was  a  change  in  color  of  the  velocity  vector  on  the  HUD  a  useful 
indication  of  abnormal  and  emergency  conditions  in  the  color  format  presentations?  If 
not,  how  would  you  indicate  this  information  on  the  HUD? 

Yes,  it  was  adequate  (8  pilots). 

No,  blink  it,  too  (6  pilots). 


No,  add  something  else  (6  pilots). 

Do  not  need  heading  in  all  three  displays  (1  pilot). 


Question  11.  Do  you  think  the  waypoint  and  target  flags  on  the  HUD  and  on  the 
Vertical  Situation  Display  are  useful  cues?  Were  these  cues  useful  in  crosschecking 
flight  progress  on  the  Horizontal  Situation  Display? 

Yes,  they  are  useful  (16  pilots). 

Good  on  VSD,  but  not  HUD  (2  pilots). 

No,  unless  they  were  real  sensor  input  (1  pilot). 

No,  HSD  indication  is  enough  (1  pilot). 


Question  12.  What  is  your  general  feeling  about  the  river  and  highway  presentations 
on  the  low  altitude  Vertical  Situation  Display? 

Not  useful  (9  pilots). 

Might  be  useful  in  low  altitude  VMC  navigation  (3  pilots). 

Good  (3  pilots).  Good  but  not  in  monochromatic  VSD  (2  pilots). 


Question  13.  What  is  your  general  feeling  about  the  usefulness  of  the  target  expansion 
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feature  on  the  map  display? 


Useful  (14  pilots). 

Not  useful  (2  pilots). 

E  etter  if  it  were  real  sensor  imagery  (4  pilots), 
"...greatest  feature  in  the  whole  system"  (1  pilot), 
"...one  of  the  less  important  features"  (1  pilot). 


Question  14.  What  is  your  general  feeling  about  the  usefulness  of  the  fuel  range  rings 
on  the  map  display? 

Good  (7  pilots). 

Good  but  add  fixed  range  rings  or  something  to  improve  estimation  (7  pilots). 
Good  but  add  digital  range  (5  pilots).  • 

Add  fuel  flow  gauge  (2  pilots). 

Good,  but  might  be  hard  to  implement  (1  pilot). 

Question  15.  What  is  your  general  feeling  about  the  usefulness  of  the  air-to-air 
pictorial  format  on  the  horizontal  situation  display? 

Good  (9  pilots). 

Good  but  makes  assumptions  about  adversary's  ordinance  (5  pilots). 

Good  but  add  target  designator  box  on  HUD  (2  pilots). 

Good  but  may  need  to  be  reduced  for  multiplane  engagements  (1  pilot). 

Good,  but  need  more  range  (1  pilot). 

No  opinion  (2  pilots). 

Question  16.  What  did  you  like  most  about  the  pictorial  formats  that  were  presented 
in  this  simulation? 

Speed  of  understanding  malfunctions  and  corrective  actions  with  pictorial 

checklists  (10  pilots). - 

HSD  (5  pilots).  - 

Threat  and  terrain  depiction  (4  pilots). 

Color  fill  (2  pilots), 
f  'se  of  color  (2  pilots). 

Stores  formats  (2  pilots). 

Versatility  of  format  creation  (1  pilot). 

Emergency  indication  on  HUD  (1  pilot). 


Clear,  sharp  pictures  (1  pilot). 

Fuel  format  (1  pilot). 

VSD  and  HSD  (1  pilot). 

Air-to-air  mode  (1  pilot). 

Question  17.  What  did  you  dislike  most  about  the  pictorial  formats  that  were 
presented  in  this  simulation? 

Fuel  and  engine  displays  (2  pilots). 

,  HUD  too  cluttered  (2  pilots). 

HUD  terrain  presentation  (1  pilot). 

HUD  pathway  (1  pilot). 

VSD  -  the  information  should  be  on  the  HUD  (1  pilot). 

Flying  the  pathway  and  getting  used  to  it  (1  pilot). 

Too  much  non-useful  information  (1  pilot). 

Difficulty  of  seeing  trends  in  status  displays  (1  pilot). 

Cartoon  advisories.  Should  have  actions  memorized  (1  pilot). 

Need  accurate  thrust  setting  information  on  engine  format  (1  pilot). 

Fuel,  engine  and  electrical  status  displays  (1  pilot). 

Fuel  display  (1  pilot). 

Want  North-up  HSD  (1  pilot). 

Color  fill  on  HUD  (1  pilot). 

Color  line  and  monochrome  display  modes  (1  pilot). 

Monochrome  mode  (1  pilot). 

Question  18.  Would  you  modify  the  content  or  layout  of  any  of  the  pictorial  formats 
used  in  this  simulation?  If  so,  what  would  you  change? 

Head  Up  Display 

Add  steering  symbology  back  to  pathway  and  for  turns  (7  pilots). 

Cut  out  clutter  in  low  attitude  HUD  (5  pilots). 


Add  threat  indication  to  HUD  (4  pilots). 

Replace  channel  with  fly-to  symbol  and  single  line  (2  pilots) 
Add  better  artificial  horizon  (2  pilots). 

Add  pitch  ladder  and  roll  indicator  (2  pilots). 

Need  better  altitude  information  (2  pilots). 

Eliminate  artificial  terrain  (2  pilots). 

Integrate  nose  camera  video  (1  pilot). 

Add  vertical  speed  (1  pilot). 

Add  air-to-air  threat  information  (1  pilot). 

Make  it  like  A7E  HUD  (1  pilot). 

Add  declutter  capability  (1  pilot). 

Make  heading,  airspeed  arid  attitude  boxes  larger  (1  pilot). 
Need  range  information  to  terrain  features  (1  pilot). 

Need  low  altitude  warning  (1  pilot). 

Make  lock-on  X  larger  in  monochrome  (1  pilot). 

What  would  weather  look  like  (1  pilot)? 

Good  as  is  (1  pilot). 

Color  line  best  (1  pilot). 

Monochrome  worst  (1  pilot). 

Monochrome  best  (1  pilot). 

Vertical  Situation  Display 

Good  as  is  (8  pilots). 

VSD  redundant  except  for  threats  (4  pilots). 

Make  low  altitude  VSD  available  full  time  (1  pilot). 

Make  high  altitude  version  like  A6  VDI  (1  pilot). 

High  altitude  -  make  roll  index  bigger  (1  pilot). 

Needs  roll,  attitude  and  steering  information  (1  pilot). 

Add  radar  terrain  imagery  (1  pilot). 

Simple  heading  tape  better  than  single  box  (1  pilot). 


Move  ground  texture  to  indicate  ground  motion  (1  pilot). 

What  would  weather  look  like  (1  pilot)? 

Exchange  locations  of  VSD  and  HSD  (1  pilot). 

Use  color  fill  (2  pilots). 

Use  color  line  (1  pilot). 

Monochrome  worst  (2  pilots). 

Horizontal  Situation  Display 

(Good  (3  pilots). 

W'sh  I  had  it  in  A6  (1  pilot). 

Air-to-air  makes  assumptions  about  adversary's  characteristics  (2  pilots). 
Provide  declutter  for  air-to-air  (1  pilot). 

Air-to-air  range  should  be  selectable  (1  pilot). 

Add  velocity,  identification  and  probable  armament  load  to  air-to-air  mode 
(1  pilot). 

Add  offset-to-bottom  option  (2  pilots). 

Add  real  imagery  (1  pilot). 

State  range  as  radius  rather  than  display  height  (1  pilot). 

Add  heading  on  each  leg  (1  pilot). 

Add  range  rings  (1  pilot). 

Add  North-up  option  (1  pilot). 

HSD  shotild  not  have  time  critical  information  (1  pilot). 

What  would  weather  look  like  (1  pilot)? 

Color  fill  best  (3  pilots). 

Color  versions  better  (2  pilots). 

Stores  Status  and  Stores  Program  MPD  Formats 

Good  (13  pilots). 

Master  arm  switch  is  a  problem  (3  pilots). 

Programming  easy  but  eliminate  entry  key  (1  pilot). 
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Color  fill  best  (4  pilots). 

Color  line  and  color  fill  better  (2  pilots). 

Engine  Status  MPD  Format 

Good  (4  pilots). 

Throw  it  out  (1  pilot). 

Too  cluttered  (3  pilots). 

Too  simple  (1  pilot). 

Too  much  of  a  cartoon  (1  pilot). 

Improve  power  setting  readings  <9  pilots). 

Add  numeric  RPM  (8  pilots). 

Add  numeric  EGT  (4  pilots). 

Add  numeric  fuel  flow  (3  pilots). 

Add  oil  pressure  and  quantity  (1  pilot). 

Color  fill  best  (1  pilot). 

Color  line  best  (1  pilot). 

Hydraulic  Status  MPD  Format 

Good  (9  pilots). 

Terrible  (1  pilot). 

Could  replace  with  gauge  (3  pilots). 

Need  numerical  information  (3  pilots). 

Combine  with  engine  format  (1  pilot). 

Need  more  detail  (1  pilot). 

Monochrome  was  all  right  (1  pilot). 

Electrical  Status  MPD  Format 

Good  (10  pilots). 

Replace  with  gauges  or  annunciator  (3  pilots). 

Oust  tell  me  in  words  (1  pilot).  , 

Hard  to  distinguish  between  temporary  and  permanent  failure  (1  pilot). 
Color  line  best  (1  pilot). 

Monochrome  wOrst  (1  pilot). 


Fuel  Status  MPD  Format 


Good  (10  pilots). 

Need  numeric  fuel  flow  somewhere  (5  pilots). 

Make  valve  indications  clearer  (2  pilots). 

Add  numeric  total  fuel  to  format  (2  pilots). 

Need  more  complete  schematic  (1  pilot). 

Prefer  digital  indication  by  tank  (1  pilot). 

Boost  pump  failure  indication  should  be  clearer  (1  pilot). 

Replace  with  annunciators  (1  pilot). 

Monochromatic  worst  (1  pilot). 

Emergency  and  Advisory  MPD  formats 

Good  (9  pilots). 

Prefer  checklists  1 7  pilots). 

Pilot  should  already  know  this  information  (9  pilots). 

Saves  memorization  (1  pilot). 

Add  warning  light  on  VSD  (1  pilot). 

Make  each  step  a  separate  presentation  (1  pilot). 

Should  come  up  automatically  (1  pilot). 

Airplane  this  smart  should  be  able  to  fix  these  problems  and  just  announce 
fix  (1  pilot). 

Borders  too  bright  (1  pilot). 

M  'tochrome  worst  (1  pilot). 

Color  made  little  difference  (1  pilot). 

Question  19.  Do  you  have  any  suggestions  for  additional  pictorial  formats  that  might 
be  developed? 

No  additional  (3  pilots). 

Geez,  no  (1  pilot)! 


Communications  and  navigation  (2  pitots). 

Put  VSD  threat  information  in  HUD  (1  pitot). 

Emergency  pull-up  indication  in  HUD  (2  pilots). 

VSI  in  HUD  (1  pilot). 

Integrate  actual  pictures  in  HUD  (I  pilot). 

Variable  accuracy  requirement  in  pathway  to  reflect  operational 
requirement  (1  pilot). 

Better  steering  and  attitude  in  HUD  (1  pilot). 

Make  HUD  altimeter  a  tape  (1  pilot). 

Bank  scale  on  HUD  and  VSD  (1  pilot). 

Air-to-air  display  on  VSD  (1  pilot). 

Sensor  pictures  on  VSD  (1  pilot). 

Compass  rose  on  VSD  or  HSD  (1  pilot). 

Format  to  steer  for  stem  engagement  -  also  useful  for  refuelling 
(1  pilot). 

Straight  course  and  distance  for  emergency  return  to  base  (1  pilot) 
HSD  scale  should  read  radius  (1  pilot). 

Compass  card  with  TACAN,  ILS  and  flight  director  (1  pilot). 
Offcenter  airplane  to  bottom  of  HSD  (1  pilot). 

Pictorial  evasion  format,  chaff.  ^CM,  and  maneuver  (1  pilot). 
Engine  and  hydraulic  oil  pressure  and  quantity  (1  pilot). 

Fuel  flow  (1  pilot). 

More  checklists  (1  pilot). 

RHAW  display  (1  pilot). 

Include  ground  speed  for  low  level,  airspeed  and  Mach  number  at 
altitude  (1  pilot). 

Ordered  single  action  pictures  for  advisories  (1  pilot). 

Clean  up  engine  format  (1  pilot). 

Environmental  system  (1  pilot). 

Voice  actuated  communications  change  (1  pilot). 

Flash  master  caution  (1  pilot). 


APPENDIX  C 

PILOTS'  QUESTIONNAIRE  -  THREAT  WARNING  STUDY 

The  pages  which  follow  contain  a  copy  of  the  questionnaire  each  pilot  filled  out  at  the 
completion  of  their  data  taking  flights  in  the  simulator.  The  numbers  in  the  cells  are 
the  mean  responses  across  the  twelve  subjects. 
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1.  The  purpose  of  this  questionnaire  is  to  elicit  your  opinions  of  the 
symbology  used  in  this  study.  Threat  warning  elements  are  emphasized,  but 
ratings  are  also  requested  for  other  display  symbology.  Defined  below  are 
attributes  or  characteristics  of  display  symbology. 

Usability  in  Color.  How  usable  was  this  display  element  in  the  color 
display  mode? 

Conspicuousness  in  Color.  How  easy  was  it  to  see  this  display 
element  in  the  color  mode? 

Usability  in  Monochrome.  How  usable  was  this  display  element  in  the 
monochrome  display  mode? 

Conspicuousness  in  Monochrome.  How  easy  was  it  to  see  this  display 
element  in  the  monochrome  display  mode? 

Location.  Is  this  format  element  in  the  right  place  and  on  the  right 
display  (HUD,  VSD,  HSD)? 


Meaning.  How  clear  or  obvious  Is  the  meaning  of  this  format  element? 

Precision.  Does  this  format  element  convey  its  information  with  the 
appropriate  level  of  precision? 

Timeliness.  Is  this  format  element  available  to  you  at  the  right 
time  and  for  the  right  Ouratlon? 

Training.  How  easily  could  this  format  element  be  learned? 


Workload.  Does  this  format  element  contribute  to  workload  or  relieve 


This  symbology 
element  Is 


Very  Good 
Moderately  Good 
Slightly  Good 
Neutral 
Slightly  Poor 
Moderately  Poor 
Very  Poor 


with  respect  to  this  attribute. 


*  IN 
.  M0N0- 
IN  COLOR  /CHROME 


HUD  SYMBOLOGY 
ELEMENTS 

Search  mode  threat 
alert  symbol 

Track  mode  threat 
alert  symbol 

Tlme-to-mlsslle 
Impact  readout 

Threat  azimuth  vector 

Pathway  In  the  sky 

Transitional  flight  director 

Terrain  Information 

Weapon  release  cue 

Waypoint/target  flags 


7  // 

f i/i/s/i 


#  /MODE  INDEPENDENT  ATTRIBUTES 

nf/z/M/z/// 


f/fli/n/s/f/t/s/f/s. 


1.751.33  1.91 

1.50 

1.42  2.41  2.21.75  1.67  2.27 

2.00  1.42  2.08 

1.58 

1.33  2.67  2.33  1.751.83  3.18 

1.25  1.67  1.42 

2.00 

1.42  1.58  1.42  1.67  1.33  1.45 

2.5  3.27  2.83 

4.36 

2.67  2.17  2.25  2.00  2.50  3.09 

3.33  1.67  3.50 

2.25 

1.58  2.3-  3.25  2.82  2.55  3.55 

4.50  3.83  5.08 

4.58 

2.67  3.5?  5k 08  3.58  4.334.36 

2.00  2,08  2.8^ 

3.0E 

1.67  2.42  2.75  1.91  1,91  2.09 

1.83  2.33  2.42 

4.0C 

1.83  1.3'  1.67  1.50  1.42  1.55 

2.08  1.83  2.08 

2.5C 

1.5*  1.6:  1.58,.  1.25  1.27  1.82 

This  symbology 
element  is 


,  1  Very  Good 

2  Moderately  Good 

3  Slightly  Good 

4  Neutral 

5  Slightly  Poor 

6  Moderately  Poor 
v  7  Very  Poor 

!l N  COLOR 


with  respect  to  this  attribute. 


VSD  SYMBOLOGY 

ELEMENTS 

Inactive  threat  lethality 
envelopes  (outlines  only) 

Active  threat  lethality 
envelopes 

Track  mode  "tractor  beam" 

Launch  mode  flashing  tractor 
beam  and  jewel  light 

Countermeasures  effectiveness 
coding  (circle  around  aircraft 
symbol) 

Ribbon  In  the  sky  (VSO  pathway) 

Above  and  below  altitude  terrain 

coding 

Geographic  and  cultural  features 

Waypoint/target  flags 


MODE  INDEPENDENT  ATTRIBUTE 


£>/&/$  /  & 
.*  /  /& /&  /&  Af  A?  /# 


2.58 

2.67 

3.17 

3.58 

1.83 

2.58 

1.8- 

1.55 

1.73 

f — .  ... 

2.18 

1.92 

1.92 

2.67 

2.83 

1.92 

2.33 

1.82 

1.64 

1.91 

2.73 

2.50 

2.50 

3.00 

3.58 

2.33 

1.75 

2.17 

1.75 

1.82 

2.55 

3.00 

2.92 

3.25 

3.83 

2.38 

2.58 

2.12 

1.67 

2.09 

2.73 

2.58 

2.67 

3.17 

3.42 

2.50 

1.83 

1.71 

1.58 

1.82 

2.95 

2.92 

2.17 

3.25 

2.50 

2.17 

2.25 

2.5C 

2.00 

3.27 

4.18 

3.33 

2.83 

4.00 

3.83 

2.08 

2.42 

2.17 

1.82 

2.82 

3.09 

2.17 

2.33 

2.83 

3.50 

1.92 

2.17 

1.82 

1.73 

2.0C 

2.45 

2.83 

2.33 

3.00 

2.92 

2.00 

1.92 

1.5£ 

1.45 

1.7: 

2.91 
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This  symbology 
element  is 


Very  Good 
Moderately  Good 
Slightly  Good 
Neutral  " 

Slightly  Poor  with  respect  to  this  attribute. 

Moderately  Poor 

Very  Poor  J  IN 

M0N0- 


HSD  SYMBOLOGY 

ELEMENTS 

Inactive  threat  lethality 
envelopes  (outline  only) 

Active  threat  lethality 
envelopes 

Tractor  beam 

Missile  Position 

Bogie  number  and  type 
information  (A/A) 

Bogie  altitude  difference 
information  (A/A) 

Above  and  below  altitude 

terrain  coding 

Geographical  and  cultural 
features 


2.  This  part  of  the  questionnaire  is  designed  to  elicit  your  opinions  on  the 
relative  usability  of  the  two  display  presentation  modes  -  monochromatic 
and  color.  Please  place  a  number  on  each  line  to  indicate,  for 
monochromatic  and  for  color,  the  degree  of  ease  or  difficulty  of 
understanding  the  particular  display  information. 

Easy  Difficult 


Very 

1 


Moderately 

2 


Slightly  Neutral 

3  4 


Slightly  Moderately  Very 
5  6  7 


Monochrome  ,  Color 


1.  Interpretation  of  threat  alert  symbology 

on  the  HUD  was:  1.92 

2.  Interpretation  of  HUD  pathway  information  was:  3.67 

3.  Interpretation  of  HliD  terrain  information  was:  3.42 

4.  Interpretation  of  surface-to-air  threat  mode 

information  on  the  VSD  was:  2.83 

5.  Interpretation  of  surface-to-air  threat  type 

information  on  the  VSD  was:  3.50 


6.  Interpretation  of  countermeasures  effectiveness 

information  on  the  VSD  was:  2.75 


1.75 

2.83 

2.25 

2.25 

2.67 

1.83 


7.  Interpretation  of  the  VSD  ribbon  in  the  sky  was:  3.00 


2.42 


.8.  Interpretation  of  VSD  terrain  Information  was:  2.67  1.67 
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Easy 


Difficult 


Very  Moderately  Sligntly  Neutral  Slightly  Moderately  Very 
1  2  3  4  5  6  7 


-  , 

Monochrome 

Color 

9.  Interpretation  of  surface-to-air  threat 
mode  information  on  the  HSD  was: 

2.58 

2.33 

10.  Interpretation  of  surfaco-to-atr  threat 
type  information  on  the  HSO  was: 

3.75 

3.08 

11.  Interpretation  of  countermeasures  effectiveness 

information  on  the  HSD  was: 

3.16 

2.08 

12.  Interpretation  of  current  missile  position 

information  on  the  HSD  was: 

2.75 

2.41 

13.  Interpretation  of  A/A  threat  mode  Information 
on  the  HSD  A/A  format  was: 

2.00 

1.75 

14.  Interpretation  of  A/A  threat  type  information 
on  the  HSD  A/A  format  was : 

1.42 

1.42 

15.  Crosschecking  threat  mode  Information  on  the 

VSD  and  HSD  was: 

4.25 

3.^3 

16.  Crosschecking  threat  type  Information  on  the 

VSD  and  HSD  was: 

4.17 

3.58 

17.  Crosschecking  flight  path  Information  on  the 
HUD,  VSD  and  HSD  was: 

3.50 

3.33 

18.  Crosschecking  terrain  Information  on  the 

HUD,  VSD  and  HSD  was: 

3.83 

3.08 

19.  Stores  manipulation  and  status 

4.17. 

4.08 
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APPENDIX  D 


SYNOPSIS  OF  TWELVE  RESPONSES  TO  EACH  OF  TEN  OPEN-ENDED  QUESTIONS 

THREAT  WARNING  STUDY 

The  pilots  spoke  their  responses  to  the  open-ended  questions  into  a  tape  recorder. 
These  responses  were  transcribed  and  are  summarized  in  the  pages  which  follow.  As 
might  be  expected,  there  are  some  agreements  and  a  number  of  differences.  Several 
pilots  went  through  the  listed  questions  and  found  themselves  will  still  more  to  say. 
These  additional  statements  are  summarized  under  "General  Comments"  at  the  end  of 
this  appendix.  The  revised  formats  include  attempts  to  address  plurality  opinions. 


Question  1.  What  is  your  general  opinion  of  the  threat  alerting  information  on  the 
HUD?  Any  suggestions  for  change? 

Good  (7  pilots). 

Use  different  symbol  for  AAA  and  SAM  (6  pilots). 

Need  countermeasures  effectiveness  information  on  HUD  (4  pilots). 

For  AAA,  put  three  A's  instead  of  3  O's  in  the  countdown  box  (2  pilots). 

Need  continuous  search  or  track  information  in  HUD  as  well  as  alerts  for  new 
threat*  (2  pilots) 

Keep  alerting  symbol  up  longer  than  6  sec  -  perhaps  reduce  size  (1  pilot). 

Hard  to  see  threat  azimuth  vector  (2  pilots). 

Put  azimuth  vector  on  alert  symbol  vs  airplane  symbol  (1  pilot). 

Dian't  use  threat  azimuth  on  HUD  (1  pilot).  ' 

Use  different  symbols  or  color  as  track  and  launch  (2  pilots). 

Use  different  flash  rates  for  track  and  launch  (1  pilot). 

Turn  flash  rule  around  -  flash  for  search,  solid  for  track  or  launch  (I  pilot). 

Red  alerting  symbol  especially  good  (1  pilot). 

Repeat  alert  symbols  on  VSD  (1  pilot). 

Make  launch  indication  (countdown  box)  more  conspicuous  (1  pilot). 

Add  information  on  adversary  position  for  air-to-air  in  HUD  (1  pilot). 

Add  aural  warning  for  new  threat  (1  pilot). 

Put  G-meter  in  HUD  (1  pilot). 

Monochrome  confusing  (1  pilot). 

Need  better  horizon  in  HUD  (1  pilot). 

Make  HUD  numbers  larger  (1  pilot). 

Use  heading  tape  in  HUD  (1  pilot). 

Too  much  color  fill;  in  HUD  obscures  outside  scene  (1  pilot). 

Question  2.  What  is  your  general  opinion  of  the  threat  mode  and  type  of  information 
presented  on  the  VSD?  Any  suggestion  for  changes? 

Didn’t  use  it  much  (6  pilots). 

Good  (3  pilots). 
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Good  in  color  (1  pilot). 

Bad  on  monochrome  (2  pilots). 

Put  SAM  vs  AAA  distinction  on  HUD  then  eliminate  VSD  (1  pilot). 

Repeat  HUD  threat  alert  symbols  on  V.'  (L  pilot). 

Confused  AAA  site  with  launch  "jewel  light"  (1  pilot). 

Put  all  essential  information  in  HUD-use  VSD  and  HSD  for  backup 
details  (1  pilot). 

In  two-seat  aircraft,  give  HUD  to  pilot;  VSD  and  HSD  to  back  seater  (1  pilot). 
Change  airplane  symbol  so  it  shows  roll  (1  pilot). 

Put  viewpoint  in  airplane  and  use  to  amplify  and  expand  HUD  data  (1  pilot). 

I  liked  high  altitude  mode  (simple  ADI)  (1  pilot). 

"I  wouldn't  want  to  go  flying  into  a  valley  at  night  with  that  thing.”  (1  pilot). 

Question  3.  What  is  your  general  opinion  of  the  threat  mode  and  type  information 
presented  on  the  HSD?  Any  suggestions  for  changes? 

Good  (8  pilots). 

Identify  on  HSD  which  new  threat  the  HUD  is  alerting  (3  pilots).  Takes  too  long 
to  distinguish  SAM  from  AAA  (3  pilots).  Sometimes  confused  threats  with 
mountains  (3  pilots). 

Consider  eliminating  terrain  data  (3  pilots). 

Jewel  light  is  worthless  —  too  much  like  AAA  (1  pilot). 

Countermeasures  effectiveness  hard  in  monochrome  (1  pilot). 

Monochrome  tough  (1  pilot). 

Cluttered  with  all  those  threats  (1  pilot). 

Put  in  single  button  for  each  range  rather  than  stepping  through  (1  pilot). 

Question  4.  Which  display  did  you  find  most  useful  for  threat  information? 

HUD  ( 9  pilots). 

HSD  (6  pilots). 

Air-to-Air  HSD  (2  pilots). 


VSD  good  backup  (4  pilots). 

VSD  of  limited  value  (2  pilots). 

Put  more  in  HUD  so  I  don't  have  to  look  down  (3  pilots). 

Consider  mode  buttons  which  would  put  VSD  and  HSD  pictures  on  HUD  (1  pilot) 

Question  5.  Was  the  threat  symbology  on  the  HUD  adequate  for  alerting  you  to  threat 
mode  changes? 

Yes  (8  pilots). 

Add  countermeasures  effectiveness  information  to  HUD  (3  pilots). 

Retain  alert  symbol  longer  than  6  seconds.  Shrink  "old"  ones  to  reset  for  "new" 
alerts  (3  pilots). 

Consider  different  flash  rates  for  track  and  launch  (3  pilots). 

Add  threat  type  and  clear  track/launch  distinction  to  HUD  (1  pilot). 

Add  audio  tone  for  threat  warning  (2  pilots). 

Add  threat  location  vector  to  HUD  for  surface-to-air  as  well  as  air-to-air 
(1  pilot). 

Need  different  symbols  for  SAM  and  AAA  (1  pilot). 

Question  6.  Was  the  distribution  of  threat  alert  symbology  across  the  HUD,  VSD  and 
HSD  appropriate?  What  would  you  change? 

The  distribution  was  good  (3  pilots). 

Do  not  add  more  to  HUD  (1  pilot). 

More  in  HUD  (7  pilots). 

Azimuth  in  HUD  earlier  than  launch  for  surface-to-air  as  well  as  air-to-air 
(4  pilots). 

SAM/AAA  distinction  in  HUD  (2  pilots). 

Countermeasures  effectiveness  in  HUD  (2  pilots). 

Track/launch  distinction  in  HUD  (1  pilot). 

Maybe  change  scales  to  add  more  to  HUD  (1  pilot). 


Repeat  alert  in  VSD  as  it  is  in  HUD  (2  pilots). 

Improve  attitude  information  in  VSD  (1  pilot). 

Question  7.  Would  you  modify  the  layout  or  content  of  any  of  the  pictorial  formats 
used  in  this  simulation?  If  so,  what  would  you  change? 

Pitch  ’der  and  better  flight  director  in  HUD  (5  pilots). 

Put  mo*  ••  information  in  the  HUD  (3  pilots). 

Heading  tape  in  HUD  (2  pilots). 

HUD  weapon  delivery  X  hard  to  see  in  monochrome  (2  pilots). 

Altitude  tape  in  HUD  (1  pilot). 

Put  selected  weapon  information  in  HUD  (1  pilot). 

In  HUD,  get  rid  of  data  boxes  and  make  airspeed,  heading  and  airspeed  numbers 
larger  (1  pilot). 

Eliminate  waypoint  and  target  flags  from  HUD  (1  pilot). 

Remove  sides  from  HUD  pathway  (I  pilot). 

Put  countermeasures  effectiveness  in  HUD  (1  pilot). 

Put  flight  director  and  attitude  information  in  VSD  (1  pilot). 

Didn't  use  VSD  ribbon  much  (1  pilot). 

Have  VSD  revert  to  ADI  format  at  high  bank  angles  (1  pilot). 

Remove  VSD  and  move  HSD  up  (1  pilot). 

Swap  positions  of  HSD  and  VSD  (1  pilot). 

Improve  stores  management  and  status  displays  -  like  A-10  (2.  pilots). 

Hard  to  tell  mountains  from  AAA  sites  (1  pilot). 

Monochrome  displays  appeared  to  cluttered  (2  pilots). 

Put  symbol  dictionary  on  MF  u  -  then  you  could  make  any  symbol  on  other 
displays  and  look  up  its  meaning  (1  pilot). 


Question  8.  Do  you  have  any  suggestions  for  additional  pictorial  formats  that  might  be 
developed?  Specifics? 

Good  as  they  are  (3  pilots). 

Expand  stores/attack  symbology  to  include  other  tactics  and  delivery 
options  (3  pilots). 

Need  good  transitional  flight  director  -  and  ability  to  chose  its  duration 
and  mode  (1  pilot). 

Add  accelerometer  to  HUD  (1  pilot). 

Use  ribbon  vs  pathway  in  HUD  (1  pilot). 

Put  desired  as  well  as  actual  heading,  airspeed  and  altitude  in  HUD  (1  pilot). 

Add  threat  position  to  HUD  (1  pilot). 

Add  vertical  speed  to  VSD  (1  pilot). 

Add  angle-of-attack  to  VSD.  (1  pilot). 

With  more  than  3  types  of  surface  threats,  you'll  run  out  of  symbols  so  use 
alphanumerics  (1  pilot). 

Unclutter  displays  in  monochrome  (1  pilot). 

Keep  it  simple  (1  pilot). 

Question  9.  How  does  the  threat  warning  symbology  used  in  this  study  compare  with 
other  threat  warning  systems  you  may  have  used  or  know  about? 

A  lot  more  information  here.  This  is  much  better  (6  pilots). 

No  comment  -  not  familiar  with  other  systems  (5  pilots). 

Retain  aural  warning  from  older  systems  (1  pilot). 

Question  10.  How  necessary  are  the  various  threat  warning  elements  shown  in  this 
simulation?  Are  there  some  you  could  do  without  in  a  mission  of  the  sort  simulated? 

Need  it  all  (8  pilots). 

Wouldn't  trust  countermeasures  effectiveness  (3  pilots). 

Clear  up  A AA/SAM  distinction  on  HUD  (1  pilot). 


Remove  the  VSD  (2  pilots). 

Need  good  ADI  (1  pilot). 

Weapons  switchology  too  complicated  (1  pilot). 

Engine  status  poor  (1  pilot). 

Color  much  better  -monochrome  appears  cluttered  (1  pilot). 

General  Comments 

Find  way  to  integrate  more  threat  information  in  HUD  (3  pilots). 

HUD  inverted  T  difficult  to  use  (1  pilot). 

Put  angle  of  attack  on  HUD  and  VSD  (1  pilot). 

Use  something  like  Kaiser  VDI  in  A-6  rather  than  this  perspective  pictorial 
VSD  (1  pilot). 

VSD  airplane  symbol  should  rotate  about  its  center  of  gravity  (1  pilot). 

Give  VSD  and  HSD  to  the  back  seater  in  a  two-place  airplane  (1  pilot). 

Bogie  altitude  notation  in  air-to-air  HSD  -  hard  to  know  units  (1  pilot). 

Color  coding  of  countermeasures  effectiveness  was  confusing  (1  pilot). 

Need  transparent  envelope  notation  for  active  threats  for  case  when  the  threat 
is  the  target  ( 1  pilot). 
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